checking protocols.

[DL Neil]

Avi
> 
> Haven't noticed an answer to this. Did I miss anything?
> 
> I never saw the original message appear and thus expected no replies. I see a later post by Chris indicating he did not see it either. I assumed perhaps a moderator needed to approve it.

=a silly question on my part. A quick check of the list archive shows 
your original post but no previous responses.


> My question really boils down to being intrigued how the python language can be extended using documented protocols in more ways and having a place where you can see a list of the ones out there and a way to join in with your own classes reliably and so on.

=if I am interpreting your intent correctly (equally that of Pythonic 
thinking) the idea is for 'more' to happen at compile-time. This 
expectation likely comes?came from our earlier training in more 
strict/rigid languages. Whereas many characteristics of Python flow 
directly as consequences of it being (chosen/designed to be) a dynamic 
language - and thus many?most such decisions delayed until run-time.

=The first (related) example which springs to mind is object 
polymorphism. It must be so much harder to achieve such incredible power 
and flexibility (as Python does) when 'rules' are 'cast in stone' early.

=However, it might be worth making suggestions generated 'here' to the 
(hard-working) docs team. There are discussions of specific protocols, 
in their individual locations throughout the manual/library. Is it 
sufficiently clear what is expected when building our own classes 
according to 'the' protocol?

=Rather than collecting all the protocols' documentation together (cf 
the sequence currently used), maybe an cross-index somewhere?


> I note there may be many unpublished protocols used internally within modules that also create similar structures but do not expect to be emulated.

=coming from Python, the PSL, or third-parties?


> I see interpolated comments below and will interpolate within that but it gets confusing.

=am using line-prefix to help differentiate. Interleaved comments help 
(?me) with following the conversational flow - and ensuring that each 
point/question is addressed (he hopes)


> On 20/01/19 11:07 AM, Avi Gross wrote:
>> Short question. Checking if a protocol is set up?
> 
> =do you mean that to check/require that a class exhibits a particular protocol we should use abstract classes - will not instantiate unless all the required components are coded?
> 
> ========== I am not looking for any specific options but indeed including an abstract class might be one way to go. It might supply a reasonable way to ask if the class you make yourself INTENDS on running that protocol and would enforce the presence of the required methods somewhere in the inheritance chain. Of course the functions could be stubs that don't do the right thing, or anything passable.
... (previous content removed, as no longer necessary)


=The phrase "we're all adults here" is often used to justify Python's 
apparent inexactitude (cf other languages*). However, I often wonder 
whether that is merely an excuse to avoid explaining oneself properly. 
For example, 'here' (or is it another list, strictly speaking?) I read 
ANNouncements of package updates. How many tell me wonderful additions 
to the package, how much work went into them, what I'd now be able to do 
with it, and who I should credit for all such goodness - but don't 
actually get around to mentioning the package's objectives or 
applicability? Unless I know said package, these ANNs are a waste of my 
reading-time!

=On the other hand, such 'openness' as non-enforced "private variables" 
(applying the 'adults' phrase) enables us to consider them "private" in 
the normal course, but to abuse the notion should the need take us (and 
caution have been thrown to the wind)!

=Back to protocols: and if I want to code certain magic methods, etc, 
comprising a protocol, should I be allowed to choose to ignore other(s)? 
Be it on my own head!

* comparing Python to other languages is (IMHO) neither a reliable nor 
particularly logical argument. How can we be 'the best' if all we do is 
copy and reproduce everyone else - and would that even be possible? 
Surely the valid comparison is between the problems which must be solved 
and the facilities of the language chosen to code the solution?

=That said, there is no question that Python has been developing 
somewhat schizoid tendencies. Another mantra has always been: 'it is 
better to ask forgiveness than permission'*, and thus we should wrap 
assumptive code in a try...except block. This change took a long time to 
sit comfortably in my (albeit small) brain - and possibly still requires 
conscious consideration. My thought pattern** is still: first check the 
data, then ..., ie ensure denominator is not zero before dividing rather 
than

try:
     divide
except (whatever):
     print( "Not by zero, bub!" )

* (originally attributed to Adm Grace Hopper of COBOL fame, if you're 
into irony!).

** after reaching for my old man's walking stick, I'll note that far 
fewer ComSc courses these days offer Numerical Analysis, than were 
available, even expected, in 'the good, old days'. Perhaps that's why 
earlier this week I had one of our brave, young, experts puzzled by 
floating-point rounding and binary approximations?

=That said, look at all the isinstance()-type functions which have 
crept-in over the years, to say nothing of 'introspection' facilities, 
eg https://docs.python.org/3/library/inspect.html which are being ?mis 
used in creative ways. NB such appears only a few pages 'on' from the 
abcs mentioned last time.
(BTW That section of the library: "Python Runtime Services" offers much 
applicable to these and earlier points)

=NB I'm not complaining about a lack of consistency, and definitely not 
language "purity", just-saying...


=After all that, if someone gives you code, should you be able to rely 
upon it?

=Once that question, and its implications, has been considered we can 
move on to a slight rewording: if someone (else) gives you code, should 
you rely upon it?
(remember all the jokes giving neophyte PC-/MS-DOS users a single line 
of code which caused them to blindly re-format the hard drive?)

=ultimately if someone gives me a class/module instructing that it be 
used as a context manager, when my code runs it will fall-over in a 
steaming heap at the with, if there is no __exit__() (for example). At 
which point I will either reach out and apply rapid, sharp, pressure to 
the end of the miscreant's nose; or I'll reconfigure that code into 
'file-13'.

=if the "stubs" mentioned are there merely to remind me of an 
implementation concern (and hopefully, how it is expected to interface 
with the whole) then isn't their implementation (or lack) on my head?


> ==========Just to be clear, yes. That is the current state of affairs. ANY CODE can be run but probably fails when it does not follow the recipe. Any of many places that expect the iteration protocol will ask for a __iter__ and either catch the error or fail. If it works, they will call __next__ as often as needed and may or may not fail well if it is not found. And, as noted, the function may throw the right error object on completion or not and that may cause the caller to fail mysteriously or be bypassed if some other error is thrown instead. What I am hoping for is OPEN to multiple solutions or to be told there is no such need. One solution would be something that can be used to exercise the function and report if the protocol is being honored. In this case, that is fairly trivial. Just try the object in one of many places the iterator protocol is used. It may be less clear how you check if the 'with' protocol is satisfied as it may not properly close whatever it is guarding like
>    closing a file or freeing a lock. If you test that you may not notice the file remains open and so on. But will your code also work if the user closed the file already on purpose?
...

=much of this discussed, above. Failures in documentation go back for as 
long as there has been programmers!

=We should now move to such considerations at TDD (Test-driven Design) - 
again, somewhat different, albeit much the same, as perhaps we were 
first taught.

=I tell you nothing in describing the typical relationship between the 
person who writes the class and the coder who wants to use it. What are 
the chances that their minds are in complete agreeance? The class-coder 
makes various assumptions. The application-coder has hopes and 
requirements. The trick with up-front (and in-house) design is to find 
these earlier in the process. However, if time passes or the class was 
provided by a third-party, a library, or whatever; that is not possible. 
So, (your question) how can we be sure?

=The TDD philosophy is that we (application writers) should first sit 
down and prepare some tests - if I push this data in, xyz should happen; 
etc. Now, if the class has come from some charlatan or incompetent (or 
plainly does something quite different or differently) we prove it 
quickly - and the other way around, the more we test, the more 
confidence we can have.

=Isn't this preferable to 'blind faith'?

=does such TDD/acceptance testing address the request: "something that 
can be used to exercise the function and report if the protocol is being 
honored"?


> ==========I appreciate the references and will follow up. Quick side note about the phrase "magic" is that I was reading up on Ipython and Jupiter Notebookss and they have yet another concept of magic based on commands starting with "%" as in "%run file" as they are not talking to core python directly but sort of a level above.

=Back in 'the good, old days' (now I'm really in-the-groove/playing a 
broken record) we often used macros and pre-processors. Isn't this 
%-stuff familiar in that context?


> ===The ideal goal is for magical features  to be invisible and magical at one level while being very clear and aboveboard in other ways. So, I would like to be able to do something that tells me clearly what some object implements as in:
> 
> from magic import protocols_detected
> if ("iteration" in protocols_detected(my_object): ...

=an interesting idea. How well might it sit within the philosophies of 
Static Typing (https://docs.python.org/3/library/typing.html)?

PEP483: https://www.python.org/dev/peps/pep-0483/
PEP484: https://www.python.org/dev/peps/pep-0484/
PEP526: https://www.python.org/dev/peps/pep-0526/


> Now obviously some simple protocols that use one dunder method may be easier to check in a simpler way. Does the object support a sorting algorithm as in your example? But does that mean defining either __lt__ or __gt__ or must it be both as well as __eq__ and perhaps more?

=this should be explained in the docs. Pick a protocol. Do you think 
such explanations/specifications (in)sufficient?


> There are virus checkers that store lists of fingerprints that can be used to look in places within a file to guess well if it contains a known virus. I am thinking of an application that stores some similar fingerprint for lots of known protocols and when asked, produces a list like the code above, or some other method. Of course an object that wants to support lots of protocols using mix-in and abstract classes might get quite complex. And, as noted, python on a deeper level lets programmers be beyond devious as the simple-looking object turns out to have all kinds of wrappers and proxies that can obfuscate the heck out of figuring out what methods actually exist and in which of many related objects they are hidden. I honestly cannot expect any such tools to be more than heuristics that can get it wrong. To be concrete, you can make a class definition that simply does not show a definition for something like __iter__ but uses descriptors and decorators and other techniques that eff
>   ectively create the darn thing indirectly at the time it is evaluated or when it is first asked for or even delay creating what is needed until needed.
> 
> The above is speculative so please don't expect me to back it up with a specific example. LOL!

=per Static Typing, above. However, your "I honestly cannot expect any 
such tools to be more than heuristics" is enough to make me wonder 
whether it is worth considering, or if the above, and similar, 
approaches are but a 'necessary cost of doing business'? It would be 
great to have though!

-- 
Regards =dn