[Idle-dev] Non-buildbot human mediated tests

Thu May 8 15:08:09 CEST 2014

Ned Deily wrote:
>> One simple example: on OS X ..
Thanks for this suggestion. I was unaware of differences in OSX
platform. I will keep this in mind when creating the human tests and
in future development.

>> I don't think it is meaningful to talk about GUI tests of IDLE without a thorough consideration of the Tk aspects.

As Terry Reedy mentions in his reply, these tests "test" whether
visual aspects appear as intended i.e. they provide a sanity check on
all the dialog windows.

>>It also might be interesting to find out howthe Tk project tests Tk and other Tk apps; perhaps something could be reused from there.

I will do this immediately. I will find out if/how it is automated.

-------------------------------

With respect to testing on different platforms, i currently have
access to Linux, a mac osx(10.5.8, which i expect to upgrade soon),
and a winxp.
If someone wants to help test on a specific os/platform, please let me know.

------------------------------
(From my initial post).

I will provide a quick summary of discussion over the past week.

>>We can also ask the tester to choose one of {Pass,Error etc}, which will also be printed to console.
In case there is an exception, it will be printed to console. The
tester will NOT be asked to pick one of {pass,fail/error} to avoid
complication.

>>1. How to pass information to htest?
All the information required to run the tests will be placed in a
single file, instead of being scattered across files - because 1. all
tests can be run together(if and when required) 2. consistency 3.
provides a clear picture of whats covered and whats not

>>possibility of ALL these tests being run together?
There should be a possibility to run all the tests together. While
automation would be good, in the worst case, a simple "click next"
should be present to iterate through all the tests. Result to be
displayed in a way similar to current unittests and with stats.

------------------------------

Saimadhav Heblikar

On 7 May 2014 08:17, Terry Reedy <tjreedy at udel.edu> wrote:
> On 5/6/2014 3:10 PM, Ned Deily wrote:
>>
>> In article
>> <CAO3PiBjvqSkBKXytLByg35H9-aHSMjT6i29Vw9cyssvhPhpECA at mail.gmail.com>,
>>   Saimadhav Heblikar <saimadhavheblikar at gmail.com> wrote:
>>>
>>> Have i missed any other aspect?
>>
>>
>> A metapoint about IDLE: it is crucial to keep in mind that IDLE is both
>> a Python application and a Tk application.  In many ways, the latter is
>> more significant than the former because, unfortunately but unavoidably,
>> there are many more significant platform-dependent differences (e.g.
>> Windows native vs X11 vs OS X native) in Tk apps than there are in
>> Python apps.  That's primarily because Tk tries to adopt
>> platform-specific behaviors and appearances to blend in with the GUI
>> standards of the platform it is running on.
>
>
>> One simple example: on OS
>>
>> X, the standard is for there to be one application-specific menu bar
>> presented at the top of the desktop screen; with Windows and X11 apps,
>> the standard is to have a menu bar at the top of application windows.
>> This has impact on IDLE's appearance to users: on Windows, if you have
>> both a shell and an edit window open, each has its own customized menu
>> bar with both visible, whereas on OS X, the single menu bar at the top
>> shows only the menu options for the window which currently has input
>> focus.
>
>
> Interesting. My dream is to have side-by-side panes in one window. This
> would mean that the one menu bar would have a change a bit according to the
> pane with focus. (If shell and editors were on different tabs in one window,
> the effect would be the same.)
>
>
>> Another example is that the menu accelerator keyboard shortcuts
>>
>> vary from platform-to-platform due to both platform conventions and, in
>> some cases, due to shortcomings in the Tk implementations.  A special
>> case is the OS X Cocoa Tk implementation, the newest and the buggiest of
>> the Tk implementations.  The versions shipped so far with OS X releases
>> have proven to have enough serious problems that we strongly recommend
>> users to not try to use them but use a newer, third-party version (like
>> ActiveTcl) instead.
>
>
> Do the OS X peculiarities affect anything other than the menu bar? Do pop-up
> dialogs, such 'about' or 'preferences' look significantly different>
>
>
>> Because of all these differences, I don't think it is meaningful to talk
>> about GUI tests of IDLE without a thorough consideration of the Tk
>> aspects.
>
>
> There are two types of GUI 'tests'. The automated tests run on the
> buildbots. Automated Gui tests run on Windows buildbots, one OS buildbot I
> believe, and maybe one *nix bot, though mostly not on on the others.
>
> Human gui tests
> http://bugs.python.org/issue18104
> are about visual appearance and response to human input. There are 20+ at
> the end of various files. But they are seldom run because a) they are
> scattered and b) many do not run properly. Issue 18104 is about fixing and
> collecting the tests involve humans. Parts that can be automated will be,
> though some redundancy is good since the Gui part of the automated tests are
> not run everywhere. These tests will have at most one window with a menu on
> the screen at any one time.
>
>
>> To be effective, any testing program will need to plan to test
>> in all three of the major environments we currently support, taking into
>> account their differences.
>
>
> Definitedly. I would be happy if you tested htest4 before I apply a version
> of it the next few days. (It needs to be in place before May 19 so Saimadhav
> can start adding to it.) Once htest.py is somewhat complete, I would like
> various people to run it on various systems.
>
>
>> It also might be interesting to find out how
>> the Tk project tests Tk and other Tk apps; perhaps something could be
>> reused from there.
>
>
> While I do not want to duplicate tests of tkinter, I plan on looking at if
> and how tkinter tests automate user interaction.
>
> --
> Terry Jan Reedy
>
>
> _______________________________________________
> IDLE-dev mailing list
> IDLE-dev at python.org
> https://mail.python.org/mailman/listinfo/idle-dev