From saimadhavheblikar at gmail.com Wed Jul 2 08:57:13 2014 From: saimadhavheblikar at gmail.com (Saimadhav Heblikar) Date: Wed, 2 Jul 2014 12:27:13 +0530 Subject: [Idle-dev] IDLE extension to integrate 3rd party checkers and analyzers In-Reply-To: <53B36C8C.1020002@udel.edu> References: <53B0B892.3020306@udel.edu> <53B36C8C.1020002@udel.edu> Message-ID: Preface: The existing approach of using subprocess(http://bugs.python.org/issue21880), did not yield good result on 3 major platforms. In this post, I am outlining an alternate approach ----------- The proposed workflow is as follows IDLE <---> Middleware Generic API <---> Middleware Checker specific API <--> Checker IDLE and middleware generic api will go into Lib/idlelib. Middleware checker specific api will go into site-package(usually a single file). Checker, depends, but currently most checkers go into site-packages anyways. So, what happens is: Whenever IDLE receives a request to check a file with a checker, it requests the middleware generic api, passing on all parameters as necessary. The middleware generic api's only function is to pass on the requests to the middleware checker specific api and return the results back to IDLE. The middleware checker specific api, will *import* the checker and do the processing. It will return the result,(after formatting if necessary). Getting the checker specific api file into the users system could be done through existing framework: pip To ensure that the checker is present on the users system, we could mark it as a dependency when uploading to PyPI. For the middleware generic api to detect existing checkers: Create a idlelib specific folder in site-packages(i.e. every middleware checker specific api file to be placed into site-packages/idlelib/ to enable detection) Installation of middleware checker api could be done via command line.(We could also support from inside IDLE, but that is a separate issue altogether) It could be something like : pip install idle-checker-x etc. My proposal will avoid 'favoritism' in the checkers supported as well as not hardcode any checker(and its interface to IDLE) into idlelib -- Regards Saimadhav Heblikar From saimadhavheblikar at gmail.com Wed Jul 2 16:23:50 2014 From: saimadhavheblikar at gmail.com (Saimadhav Heblikar) Date: Wed, 2 Jul 2014 19:53:50 +0530 Subject: [Idle-dev] IDLE extension to integrate 3rd party checkers and analyzers In-Reply-To: References: <53B0B892.3020306@udel.edu> <53B36C8C.1020002@udel.edu> Message-ID: On 2 July 2014 12:27, Saimadhav Heblikar wrote: > >Getting the checker specific api file into the users system could be >done through existing framework: pip >To ensure that the checker is present on the users system, we could >mark it as a dependency when uploading to PyPI. > Hi, As a proof of concept for the distribution part of the earlier mail, I added two dummy packages to testpypi.python.org at https://testpypi.python.org/pypi/IdleChecker_x and https://testpypi.python.org/pypi/IdleChecker_y. (They do nothing except print a single line when imported) To install, pip install -i https://testpypi.python.org/pypi IdleChecker_x IdleChecker_y In the actual versions, these files will be responsible for performing the logic, like import the checker, format it according to IDLE's needs and send back the result. They will also have the respected checker as a dependency. The middleware generic api will import these modules along with the parameters like (filename, additional args) etc. It will return the result back to IDLE/ display the result itself. -- Regards Saimadhav Heblikar From tjreedy at udel.edu Wed Jul 2 21:07:58 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 02 Jul 2014 15:07:58 -0400 Subject: [Idle-dev] IDLE extension to integrate 3rd party checkers and analyzers In-Reply-To: References: <53B0B892.3020306@udel.edu> <53B36C8C.1020002@udel.edu> Message-ID: On 7/2/2014 2:57 AM, Saimadhav Heblikar wrote: > Preface: The existing approach of using subprocess > http://bugs.python.org/issue21880 > did not yield good result on 3 major platforms. Are there problems on Mac? I have not completely given up on subprocess on Windows for other uses, but for checking code in a Text widget, I think a direct approach is better anyway. > In this post, I am outlining an alternate approach > The proposed workflow is as follows > > IDLE <---> Middleware Generic API <---> Middleware Checker specific > API <--> Checker 'Middleware Generic API' == Checker API 'Middleware Checker specific API' == set of adapters > IDLE and middleware generic api will go into Lib/idlelib. Middleware > checker specific api will go into site-package(usually a single file). > Checker, depends, but currently most checkers go into site-packages > anyways. > > So, what happens is: > Whenever IDLE receives a request to check a file with a checker, it I think requests to check the editor buffer and requests to check a path should be kept separate. I prefer to start with the first. > requests the middleware generic api, passing on all parameters as > necessary. The middleware generic api's only function is to pass on > the requests to the middleware checker specific api and return the > results back to IDLE. For editor requests, the flow might be in the other direction. > The middleware checker specific api, will *import* the checker and do > the processing. It will return the result,(after formatting if > necessary). Or have the specific api pass the specifics back to idle to use. Here is how trivial the interfacing is for pyflakes. ==================== class CW: # CheckerWindow with .write, for instance def __init__(self): # self.text = Text() (or passed in param) self.text = [] def write(self, chunk): # self.text.insert('insert', chunk) self.text.append(chunk) def get(self): # for this demo # self.text.get('1.end') or whatever return ''.join(self.text) cw = CW() # Specific interface: import and call from pyflakes import api, reporter # Call as it mightbe configured. # "api.check({code}, {fname}, reporter.Reporter({stream}, {stream}))" # where 'stream' is an object with a .write method. # Let ew be the EditorWindow being checked. Then use above with # .format(code=ew.text.get('1.end'), fname=ew.???, stream = cw) # The two args to Reporter are warning and error streams. # Errors incluce compile() SyntaxErrors, as in second example below. # A separate error stream could 'color' error text if desired. # It could also separate counting of warnings from errors api.check('a = b\n', 'tem.py', reporter.Reporter(cw, cw)) cw.write('----\n') api.check('a =\n', 'tem.py', reporter.Reporter(cw, cw)) print(cw.get()) # view result in CW >>> tem.py:1: undefined name 'b' ---- tem.py:1:4: invalid syntax a = ^ ===================================== I believe that any properly written Python-coded checker/analyzer should provide such an interface, so the input and output can be passed back and forth as Python strings without encoding to or decoding from bytes (in Py3) or writing to or reading from external files. I would like to see a revised patch on 21880 that runs pyflakes this way so we have a working example for discussion of the ui, the dialogs and output window. > Getting the checker specific api file into the users system could be > done through existing framework: pip > To ensure that the checker is present on the users system, we could > mark it as a dependency when uploading to PyPI. > > For the middleware generic api to detect existing checkers: Create a > idlelib specific folder in site-packages(i.e. every middleware checker > specific api file to be placed into site-packages/idlelib/ to enable > detection) I dislike creating lots of little 100 bytes files. This is what I like about the config.def files. > Installation of middleware checker api could be done via command > line.(We could also support from inside IDLE, but that is a separate > issue altogether) > It could be something like : pip install idle-checker-x etc. > My proposal will avoid 'favoritism' in the checkers supported as well > as not hardcode any checker(and its interface to IDLE) into idlelib As I said on email, I am willing to favor programs that provide what i consider to be a proper interface. I would like to see some research into other programs and whether they have apis anything like the one for pyflakes, or what adaptor would be needed. -- Terry Jan Reedy From tjreedy at udel.edu Thu Jul 3 06:29:25 2014 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 03 Jul 2014 00:29:25 -0400 Subject: [Idle-dev] IDLE extension to integrate 3rd party checkers and analyzers In-Reply-To: References: <53B0B892.3020306@udel.edu> <53B36C8C.1020002@udel.edu> Message-ID: On 7/2/2014 3:07 PM, Terry Reedy wrote: > On 7/2/2014 2:57 AM, Saimadhav Heblikar wrote: >> Preface: The existing approach of using subprocess >> http://bugs.python.org/issue21880 >> did not yield good result on 3 major platforms. > > Are there problems on Mac? I have not completely given up on subprocess > on Windows for other uses, The following work to print output to the shell window. import subprocess as s p = s.Popen([r'pyflakes', r'c:\programs\python34\lib\turtle.py'], stdout=s.PIPE, stderr=s.PIPE) o,e = p.communicate() for line in o.decode().splitlines(): print(line) One of us should print the args variable in your patch to see how it might be different than the above. And then try to print the equivalent of o. On installed python, started from the icon, a blank console window is flashed. It is not pleasant. I suspect it might have something to do with sys.__stdout__ in the user process being None instead of something, as with the repository version. but for checking code in a Text widget, I > think a direct approach is better anyway. > Here is how trivial the interfacing is for pyflakes. > ==================== > class CW: # CheckerWindow with .write, for instance > def __init__(self): > # self.text = Text() (or passed in param) > self.text = [] > > def write(self, chunk): > # self.text.insert('insert', chunk) > self.text.append(chunk) > def get(self): # for this demo > # self.text.get('1.end') or whatever > return ''.join(self.text) > cw = CW() > > # Specific interface: import and call > from pyflakes import api, reporter > > # Call as it mightbe configured. > # "api.check({code}, {fname}, reporter.Reporter({stream}, {stream}))" > # where 'stream' is an object with a .write method. > # Let ew be the EditorWindow being checked. Then use above with > # .format(code=ew.text.get('1.end'), fname=ew.???, stream = cw) > # The two args to Reporter are warning and error streams. > # Errors incluce compile() SyntaxErrors, as in second example below. > # A separate error stream could 'color' error text if desired. > # It could also separate counting of warnings from errors > > api.check('a = b\n', 'tem.py', reporter.Reporter(cw, cw)) > cw.write('----\n') > api.check('a =\n', 'tem.py', reporter.Reporter(cw, cw)) > > print(cw.get()) # view result in CW > >>> > tem.py:1: undefined name 'b' > ---- > tem.py:1:4: invalid syntax > a = > ^ > ===================================== > > I believe that any properly written Python-coded checker/analyzer should > provide such an interface, so the input and output can be passed back > and forth as Python strings without encoding to or decoding from bytes > (in Py3) or writing to or reading from external files. > > I would like to see a revised patch on 21880 that runs pyflakes this way > so we have a working example for discussion of the ui, the dialogs and > output window. This suggests a possible 'send' api. In Idle: from x import check # where x is defined in the checker config. check(path, code, stream, estream) path = full path or just filename from editor(can right-click goto file/line deal with as a name?) code = None for full path, code string for filename (e)steam = objects with .write (and any other methods that end up being needed) for normal and error output. For pyflakes, the imported interface file would be --- import api, reporter def check(path, code, stream, estream): report = reporter(stream, estread) if code is not None: api.check(code, path, report) else: api.checkRecursive([path], report -- Terry Jan Reedy From saimadhavheblikar at gmail.com Thu Jul 3 19:27:57 2014 From: saimadhavheblikar at gmail.com (Saimadhav Heblikar) Date: Thu, 3 Jul 2014 22:57:57 +0530 Subject: [Idle-dev] IDLE extension to integrate 3rd party checkers and analyzers In-Reply-To: References: <53B0B892.3020306@udel.edu> <53B36C8C.1020002@udel.edu> Message-ID: Hi, I combined your(Terry Reedy) for checking editor buffer, with my earlier proposal of using PyPI to distribute the adapter files. The result is: User has to pip install a checker file(assumes the user has dependency for the time being, dependency can be forced using setuptools, currently using distutils, which does not check for dependencies) The checker gets auto-detected in IDLE.(though this does not enable the checker within IDLE) The user can choose to enable it and set additional args(additional args part not yet implemented, but it is a matter of passing on the args in the adapter files) The user can run the checker from "Run" menu as necessary. I am attaching the diff file. To install the pyflakes IDLE api, pip install -i https://testpypi.python.org/pypi IDLEPyflakes In essence, the user has to just pip install the file. It is disabled by default for security reasons(?? Is this necessary? Should we enable it by default?) Please let me know if I should proceed with this "experiment". -- Regards Saimadhav Heblikar -------------- next part -------------- diff -r 8734e881c400 Lib/idlelib/Checker.py --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/Lib/idlelib/Checker.py Thu Jul 03 22:55:56 2014 +0530 @@ -0,0 +1,188 @@ +import os +from pkgutil import iter_modules +from distutils.sysconfig import get_python_lib +from importlib import import_module +import tkinter as tk +import tkinter.messagebox as tkMessageBox +from idlelib.OutputWindow import OutputWindow +from idlelib.configHandler import idleConf + +def get_checkers(): + idlecheckers_path = os.path.join(get_python_lib(), 'idlecheckers') + checkers = [c[1][12:] for c in list(iter_modules([idlecheckers_path])) if c[1].startswith('idlechecker_')] + return checkers + +def get_enabled_checkers(): + return [c for c in get_checkers() if idleConf.GetOption('checker', c, + 'enabled', type='bool')] + +def get_checker_config(checker): + if checker == '': + raise ValueError('Checker name cannot be empty') + config = {} + config['enabled'] = idleConf.GetOption('checker', checker, 'enabled', + type='bool') + config['name'] = checker + config['additional'] = idleConf.GetOption('checker', checker, + 'additional') + return config + + +class Checker: + + menudefs = [ + ('run', [None, + ('Config Checkers', '<>') + ] + ) + ] + + def __init__(self, editwin): + self.editwin = editwin + self.parent = self.editwin.top + self.update_menu() + + def config_checker_event(self, event): + dialog = self.dialog = tk.Toplevel(self.parent) + dialog.transient(self.parent) + dialog.wm_title('Config Checkers') + dialog.geometry('+%d+%d' % (self.parent.winfo_rootx() + 30, + self.parent.winfo_rooty() + 30)) + titleLabel = tk.Label(dialog, text='Configure Checkers') + self.listbox = tk.Listbox(dialog, selectmode=tk.SINGLE) + self.update_listbox() + editButton = tk.Button(dialog, text='Edit Checker', + command=self.edit_checker) + titleLabel.pack() + self.listbox.pack() + editButton.pack(side=tk.LEFT) + + def update_listbox(self): + self.listbox.delete(0, tk.END) + checkers = get_checkers() + for c in checkers: + self.listbox.insert(tk.END, c) + + def _selected_checker(self): + index = self.listbox.curselection() + if index: + checker = self.listbox.get(index) + return checker + else: + return None + + def edit_checker(self): + checker = self._selected_checker() + if checker: + ConfigCheckerDialog(self.editwin, self, checker) + else: + tkMessageBox.showerror(title='No Checker Selected', + message='Select an existing checker to edit', + parent=self.dialog) + + def update_menu(self): + menu = self.editwin.menudict.get('run') + start = menu.index('Config Checkers') + 1 + end = menu.index(tk.END) + if start < end: + menu.delete(start, end) + if start == end: + menu.delete(start) + for checker in get_enabled_checkers(): + label = 'Run {}'.format(checker) + menu.add_command(label=label, + command=lambda checker=checker:self.run_checker(checker)) + + def run_checker(self, checker): + """Run the 3rd party checker 'checker' and display result in a new + outputwindow""" + text = self.editwin.text.get('1.0', 'end') + filename = self.editwin.io.filename or 'Untitled' + additional_args = get_checker_config(checker) + module = import_module('idlecheckers.idlechecker_'+checker) + output, error = module.process(text, filename, additional_args) + self.show_result(output, error) + + def show_result(self, output, error): + """Utility method to display 'output' and 'error' in a new + OutputWindow""" + theme = idleConf.CurrentTheme() + stdout = idleConf.GetHighlight(theme, 'stdout') + stderr = idleConf.GetHighlight(theme, 'stderr') + tagdefs = {'stdout':stdout, 'stderr':stderr} + outputwindow = OutputWindow(self.editwin.flist) + for tag, cnf in tagdefs.items(): + outputwindow.text.tag_configure(tag, **cnf) + outputwindow.write(error, 'stderr') + outputwindow.write(output, 'stdout') + + +class ConfigCheckerDialog(tk.Toplevel): + + def __init__(self, editwin, _checker, checker=None): + tk.Toplevel.__init__(self, editwin.top) + self.editwin = editwin + self._checker = _checker + self.checker = checker + self.parent = parent = self.editwin.top + self.enabled = tk.StringVar(parent) + self.additional = tk.StringVar(parent) + if checker: + config = get_checker_config(checker) + self.additional.set(config['additional'] or '') + self.enabled.set(config['enabled']) + self.create_widgets() + + def create_widgets(self): + parent = self.parent + self.configure(borderwidth=5) + title = 'Edit {} checker'.format(self.checker) + self.wm_title(title) + self.geometry('+%d+%d' % (parent.winfo_rootx() + 30, + parent.winfo_rooty() + 30)) + self.transient(parent) + self.focus_set() + # frames creation + optionsFrame = tk.Frame(self) + buttonFrame = tk.Frame(self) + + # optionsFrame + additionalLabel = tk.Label(optionsFrame, text='Additional Args') + + self.additionalEntry = tk.Entry(optionsFrame, + textvariable=self.additional) + enabledCheckbutton = tk.Checkbutton(optionsFrame, + variable=self.enabled, text='Enable Checker?') + + # buttonFrame + okButton = tk.Button(buttonFrame, text='Ok', command=self.ok) + cancelButton = tk.Button(buttonFrame, text='Cancel', + command=self.cancel) + + # frames packing + optionsFrame.pack() + buttonFrame.pack() + # optionsFrame packing + additionalLabel.pack() + self.additionalEntry.pack() + enabledCheckbutton.pack() + # buttonFrame packing + okButton.pack() + cancelButton.pack() + + def ok(self): + name = self.checker + idleConf.userCfg['checker'].SetOption(name, 'enabled', + self.enabled.get()) + idleConf.userCfg['checker'].SetOption(name, 'additional', + self.additional.get()) + idleConf.userCfg['checker'].Save() + self.close() + + def cancel(self): + self.close() + + def close(self): + self._checker.update_listbox() + self._checker.update_menu() + self.destroy() diff -r 8734e881c400 Lib/idlelib/config-extensions.def --- a/Lib/idlelib/config-extensions.def Sun Jun 29 00:46:45 2014 +0200 +++ b/Lib/idlelib/config-extensions.def Thu Jul 03 22:55:56 2014 +0530 @@ -94,3 +94,9 @@ enable_shell=0 enable_editor=1 +[Checker] +enable=1 +enable_shell=0 +enable_editor=1 +[Checker_bindings] +config-checker= diff -r 8734e881c400 Lib/idlelib/configHandler.py --- a/Lib/idlelib/configHandler.py Sun Jun 29 00:46:45 2014 +0200 +++ b/Lib/idlelib/configHandler.py Thu Jul 03 22:55:56 2014 +0530 @@ -162,6 +162,7 @@ (user home dir)/.idlerc/config-extensions.cfg (user home dir)/.idlerc/config-highlight.cfg (user home dir)/.idlerc/config-keys.cfg + (user home dir)/.idlerc/config-checker.cfg """ def __init__(self): self.defaultCfg={} @@ -182,7 +183,7 @@ else: # we were exec'ed (for testing only) idleDir=os.path.abspath(sys.path[0]) userDir=self.GetUserCfgDir() - configTypes=('main','extensions','highlight','keys') + configTypes=('main','extensions','highlight','keys', 'checker') defCfgFiles={} usrCfgFiles={} for cfgType in configTypes: #build config file names @@ -282,9 +283,11 @@ Get a list of sections from either the user or default config for the given config type. configSet must be either 'user' or 'default' - configType must be one of ('main','extensions','highlight','keys') + configType must be one of ('main','extensions','highlight','keys', + 'checker') """ - if not (configType in ('main','extensions','highlight','keys')): + if not (configType in ('main','extensions','highlight','keys', + 'checker')): raise InvalidConfigType('Invalid configType specified') if configSet == 'user': cfgParser=self.userCfg[configType]