Controlling the Mac OSX GUI via Python?

[Christian Gollwitzer]

Am 01.07.16 um 01:12 schrieb Lawrence D’Oliveiro:
> On Friday, July 1, 2016 at 10:35:42 AM UTC+12, Chris Angelico wrote:
>>
>> On Fri, Jul 1, 2016 at 8:27 AM, Lawrence D’Oliveiro wrote:
>>>
>>> GUIs, by their nature, are not designed to be automated.
>>
>> Doesn't mean they can't be!
>
> It’s more trouble than it’s worth. Different (even minor) versions of
> the program will move UI elements around. Things that look and behave
> like buttons might have underlying code that is not quite the same as
> that for buttons. There will be subtle timing issues, where something
> doesn’t quite get enabled in time for your script to click on it. You
> will suffer mysterious intermittent problems, where a sequence works
> one time but not another time.

I think you misunderstood the request. Yes, simulating mouse clicks with 
fixed coordinates etc. is prone to such failure, but there are better 
methods. These mouse clicks and keyboard events usually trigger a method 
call inside the GUI program. If there are any means to call that method 
directly, you are independent from the graphics itself.

These bindings then usually call some higher-level functions. Many 
applications export their internal structure via some general 
communications mechanism. On Windows, COM is such a mechanism, CORBA is 
another one, and on the Mac that are Apple Events. AppleScript is a 
Mac-specific (strange) language which allows you to send arbitrary Apple 
events to applications. This has nothing to do with simulating mouse 
clicks. A deprecated Python package is found here:

	http://appscript.sourceforge.net/py-appscript/

If you look at the first example:
	app('TextEdit').documents['ReadMe'].paragraphs[1].get()

its immediately clear that this is on a very different level than 
simulating Copy/Paste using the mouse to get the first paragraph. 
According to my Google-Fu, howver, py-appscript stopped working with 
some OSX update. One possibility would be to send the equivalent 
applescript to the "osascript" program via subprocess.

> Again: GUIs are for manual operations to be performed by humans. For
> automation, you need a function-based scripting API and a command
> line.

Yes and no. Sometimes GUIs export an API which is just not obvious to 
the user, sometimes they have an internal structure which can be exploited.

	Christian