[Idle-dev] Non-buildbot human mediated tests

[Terry Reedy]

On 5/6/2014 3:10 PM, Ned Deily wrote:
> In article
> <CAO3PiBjvqSkBKXytLByg35H9-aHSMjT6i29Vw9cyssvhPhpECA at mail.gmail.com>,
>   Saimadhav Heblikar <saimadhavheblikar at gmail.com> wrote:
>> Have i missed any other aspect?
>
> A metapoint about IDLE: it is crucial to keep in mind that IDLE is both
> a Python application and a Tk application.  In many ways, the latter is
> more significant than the former because, unfortunately but unavoidably,
> there are many more significant platform-dependent differences (e.g.
> Windows native vs X11 vs OS X native) in Tk apps than there are in
> Python apps.  That's primarily because Tk tries to adopt
> platform-specific behaviors and appearances to blend in with the GUI
> standards of the platform it is running on.

 > One simple example: on OS
> X, the standard is for there to be one application-specific menu bar
> presented at the top of the desktop screen; with Windows and X11 apps,
> the standard is to have a menu bar at the top of application windows.
> This has impact on IDLE's appearance to users: on Windows, if you have
> both a shell and an edit window open, each has its own customized menu
> bar with both visible, whereas on OS X, the single menu bar at the top
> shows only the menu options for the window which currently has input
> focus.

Interesting. My dream is to have side-by-side panes in one window. This 
would mean that the one menu bar would have a change a bit according to 
the pane with focus. (If shell and editors were on different tabs in one 
window, the effect would be the same.)

 > Another example is that the menu accelerator keyboard shortcuts
> vary from platform-to-platform due to both platform conventions and, in
> some cases, due to shortcomings in the Tk implementations.  A special
> case is the OS X Cocoa Tk implementation, the newest and the buggiest of
> the Tk implementations.  The versions shipped so far with OS X releases
> have proven to have enough serious problems that we strongly recommend
> users to not try to use them but use a newer, third-party version (like
> ActiveTcl) instead.

Do the OS X peculiarities affect anything other than the menu bar? Do 
pop-up dialogs, such 'about' or 'preferences' look significantly different>

> Because of all these differences, I don't think it is meaningful to talk
> about GUI tests of IDLE without a thorough consideration of the Tk
> aspects.

There are two types of GUI 'tests'. The automated tests run on the 
buildbots. Automated Gui tests run on Windows buildbots, one OS buildbot 
I believe, and maybe one *nix bot, though mostly not on on the others.

Human gui tests
http://bugs.python.org/issue18104
are about visual appearance and response to human input. There are 20+ 
at the end of various files. But they are seldom run because a) they are 
scattered and b) many do not run properly. Issue 18104 is about fixing 
and collecting the tests involve humans. Parts that can be automated 
will be, though some redundancy is good since the Gui part of the 
automated tests are not run everywhere. These tests will have at most 
one window with a menu on the screen at any one time.

> To be effective, any testing program will need to plan to test
> in all three of the major environments we currently support, taking into
> account their differences.

Definitedly. I would be happy if you tested htest4 before I apply a 
version of it the next few days. (It needs to be in place before May 19 
so Saimadhav can start adding to it.) Once htest.py is somewhat 
complete, I would like various people to run it on various systems.

> It also might be interesting to find out how
> the Tk project tests Tk and other Tk apps; perhaps something could be
> reused from there.

While I do not want to duplicate tests of tkinter, I plan on looking at 
if and how tkinter tests automate user interaction.

-- 
Terry Jan Reedy