From jython-checkins at python.org Sat Apr 14 04:50:12 2018 From: jython-checkins at python.org (jeff.allen) Date: Sat, 14 Apr 2018 08:50:12 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Catch-up_on_bug-fixes_in_N?= =?utf-8?q?EWS?= Message-ID: <20180414085012.1.E1F15960806337D3@mg.python.org> https://hg.python.org/jython/rev/c595b37d54a4 changeset: 8151:c595b37d54a4 user: Jeff Allen date: Sat Mar 24 20:47:37 2018 +0000 summary: Catch-up on bug-fixes in NEWS We recently had a clear-out on bugs.jython.org, finding a number of issues that have been fixed in the course of other work (or we forgot to close them). files: NEWS | 11 ++++++++++- 1 files changed, 10 insertions(+), 1 deletions(-) diff --git a/NEWS b/NEWS --- a/NEWS +++ b/NEWS @@ -2,7 +2,15 @@ For more details, please see https://hg.python.org/jython - Developement tip +Developement tip + Bugs fixed + - [ 1866 ] Parser does not have mismatch token error messages caught by BaseRecognizer + - [ 1930 ] traceback raises exception in os.py + - [ 2419 ] List expected failures by OS platform in regrtest.py + - [ 2611 ] mkdir() operation in /Lib/os.py has different behavior when running in Docker container + - [ 2655 ] __findattr__ is final (update documentation) + - [ 2646 ] Test failure in test_os_jy after utf-8 decode + - [ 2469 ] jython launcher fails on Windows if JAVA_HOME is set - [ 2650 ] Detail message is not set on PyException from PythonInterpreter - [ 2403 ] VerifyError when implementing interfaces containing default methods (Java 8) @@ -34,6 +42,7 @@ Jython 2.7.1rc2 Bugs fixed - [ 2536 ] deadlocks in regrtests due to StackOverflowError in finally block (workaround, still open) + - [ 2348 ] site module does not import if usernames contain non-ascii characters - [ 2356 ] java.lang.IllegalArgumentException on startup on Windows if username not ASCII - [ 1839 ] sys.getfilesystemencoding() is None (now utf-8) - [ 2579 ] Pyc files are not loading for too large modules if path contains __pyclasspath__ -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Sat Apr 14 04:50:12 2018 From: jython-checkins at python.org (jeff.allen) Date: Sat, 14 Apr 2018 08:50:12 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Use_relative_path_in_launc?= =?utf-8?q?her_only_if_shorter_than_absolute_path?= Message-ID: <20180414085012.1.642E909F1CE7CFD9@mg.python.org> https://hg.python.org/jython/rev/4af36339826e changeset: 8152:4af36339826e user: Jeff Allen date: Wed Mar 28 17:25:06 2018 +0100 summary: Use relative path in launcher only if shorter than absolute path We opted for relative paths to make paths and command-lines more readable, and avoid some non-ascii encoding risks, when Jython is installed locally to a user. The choice was counter-productive when Jython was installed in a shared location. files: src/shell/jython.py | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-) diff --git a/src/shell/jython.py b/src/shell/jython.py --- a/src/shell/jython.py +++ b/src/shell/jython.py @@ -202,9 +202,9 @@ # Python 2 thinks in bytes. Carefully normalise in Unicode. path = os.path.realpath(bytes_path.decode(ENCODING)) try: - # If possible, make this relative to the CWD. - # This helps manage multi-byte names in installation location. - path = os.path.relpath(path, os.getcwdu()) + # If shorter, make this relative to the CWD. + relpath = os.path.relpath(path, os.getcwdu()) + if len(relpath) < len(path): path = relpath except ValueError: # Many reasons why this might be impossible: use an absolute path. path = os.path.abspath(path) -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Sat Apr 14 04:50:12 2018 From: jython-checkins at python.org (jeff.allen) Date: Sat, 14 Apr 2018 08:50:12 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Allow_JAVA=5FSTACK_and_JAV?= =?utf-8?q?A=5FMEM_to_be_just_the_size=2E_Fixes_=232501=2E?= Message-ID: <20180414085012.1.BBFDAAA687752AE5@mg.python.org> https://hg.python.org/jython/rev/0d91e075bb1b changeset: 8153:0d91e075bb1b user: Jeff Allen date: Wed Mar 28 20:14:32 2018 +0100 summary: Allow JAVA_STACK and JAVA_MEM to be just the size. Fixes #2501. It was unclear whether -Xss and -Xmx prefixes were required, and now they are imputed if missing. This fixes #2501 for Windows, but the Unix shell script launcher needs the same change. files: NEWS | 1 + src/shell/jython.py | 27 ++++++++++++++++++++++----- 2 files changed, 23 insertions(+), 5 deletions(-) diff --git a/NEWS b/NEWS --- a/NEWS +++ b/NEWS @@ -4,6 +4,7 @@ Developement tip Bugs fixed + - [ 2501 ] JAVA_STACK doesn't work (fixed for Windows launcher only) - [ 1866 ] Parser does not have mismatch token error messages caught by BaseRecognizer - [ 1930 ] traceback raises exception in os.py - [ 2419 ] List expected failures by OS platform in regrtest.py diff --git a/src/shell/jython.py b/src/shell/jython.py --- a/src/shell/jython.py +++ b/src/shell/jython.py @@ -48,7 +48,7 @@ def get_env(envvar, default=None): """ Return the named environment variable, decoded to Unicode.""" v = os.environ.get(envvar, default) - # Tolerate default given as bytes, as we're bound to forget sometimes + # Result may be bytes but we want unicode for the command if isinstance(v, bytes): v = v.decode(ENCODING) # Remove quotes sometimes necessary around the value @@ -56,6 +56,23 @@ v = v[1:-1] return v +def get_env_mem(envvar, default): + """ Return the named memory environment variable, decoded to Unicode. + The default should begin with -Xmx or -Xss as in the java command, + but this part will be added to the environmental value if missing. + """ + # Tolerate default given as bytes, as we're bound to forget sometimes + if isinstance(default, bytes): + default = default.decode(ENCODING) + v = os.environ.get(envvar, default) + # Result may be bytes but we want unicode for the command + if isinstance(v, bytes): + v = v.decode(ENCODING) + # Accept either a form like 16m or one like -Xmx16m + if not v.startswith(u"-X"): + v = default[:4] + v + return v + def encode_list(args, encoding=ENCODING): """ Convert list of Unicode strings to list of encoded byte strings.""" r = [] @@ -268,14 +285,14 @@ if hasattr(self.args, "mem"): return self.args.mem else: - return get_env("JAVA_MEM", "-Xmx512m") + return get_env_mem("JAVA_MEM", "-Xmx512m") @property def java_stack(self): if hasattr(self.args, "stack"): return self.args.stack else: - return os.environ.get("JAVA_STACK", "-Xss2560k") + return get_env_mem("JAVA_STACK", "-Xss2560k") @property def java_opts(self): @@ -386,9 +403,9 @@ --profile: run with the Java Interactive Profiler (http://jiprof.sf.net) -- : pass remaining arguments through to Jython Jython launcher environment variables: -JAVA_MEM : Java memory (sets via -Xmx) +JAVA_MEM : Java memory size as a java option e.g. -Xmx600m or just 600m +JAVA_STACK : Java stack size as a java option e.g. -Xss5120k or just 5120k JAVA_OPTS : options to pass directly to Java -JAVA_STACK : Java stack size (sets via -Xss) JAVA_HOME : Java installation directory JYTHON_HOME: Jython installation directory JYTHON_OPTS: default command line arguments -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Sat Apr 14 04:50:12 2018 From: jython-checkins at python.org (jeff.allen) Date: Sat, 14 Apr 2018 08:50:12 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Tolerate_looser_order_of_P?= =?utf-8?q?ython_and_Java_arguments_to_launcher=2E?= Message-ID: <20180414085012.1.56410B278C9E3FC8@mg.python.org> https://hg.python.org/jython/rev/b556dcdf8258 changeset: 8154:b556dcdf8258 user: Jeff Allen date: Wed Mar 28 20:30:13 2018 +0100 summary: Tolerate looser order of Python and Java arguments to launcher. This allows idioms like cmd = [sys.executable, '-E'].extend(args) to succeed when args includes JVM arguments (e.g. in Lib/test/script_helper.py). Currently only for jython.py launcher. files: src/shell/jython.py | 7 ++++++- 1 files changed, 6 insertions(+), 1 deletions(-) diff --git a/src/shell/jython.py b/src/shell/jython.py --- a/src/shell/jython.py +++ b/src/shell/jython.py @@ -104,6 +104,7 @@ parsed.profile = False # --profile flag given parsed.properties = OrderedDict() # properties to give the JVM parsed.java = [] # any other arguments to give the JVM + unparsed = list() it = iter(args) next(it) # ignore sys.argv[0] @@ -143,13 +144,17 @@ elif arg in (u"--boot", u"--jdb", u"--profile"): setattr(parsed, arg[2:], True) i += 1 + elif len(arg) >= 2 and arg[0] == u'-' and arg[1] in u"BEisSuvV3": + unparsed.append(arg) + i += 1 elif arg == u"--": i += 1 break else: break - return parsed, args[i:] + unparsed.extend(args[i:]) + return parsed, unparsed class JythonCommand(object): -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Sat Apr 14 04:50:13 2018 From: jython-checkins at python.org (jeff.allen) Date: Sat, 14 Apr 2018 08:50:13 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Rebuild_jython=2Eexe_follo?= =?utf-8?q?wing_jython=2Epy_changes?= Message-ID: <20180414085013.1.C2FE9C3BE6BB38DA@mg.python.org> https://hg.python.org/jython/rev/27fa5e00ea2a changeset: 8155:27fa5e00ea2a user: Jeff Allen date: Thu Mar 29 22:51:20 2018 +0100 summary: Rebuild jython.exe following jython.py changes Followed slavishly http://jython-devguide.readthedocs.io/en/latest/setup_jy.html#the-launcher-jython-exe files: src/shell/jython.exe | Bin 1 files changed, 0 insertions(+), 0 deletions(-) diff --git a/src/shell/jython.exe b/src/shell/jython.exe index 8a4abc7af726b5b52902677b534479dcc23c2361..0117670e8db2e1c9b47ca09cbed25dd70f5f1717 GIT binary patch [stripped] -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Mon Apr 16 17:44:14 2018 From: jython-checkins at python.org (jeff.allen) Date: Mon, 16 Apr 2018 21:44:14 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Refactor_PyJavaType=2Einit?= =?utf-8?q?_for_readability_=28no_material_change=29=2E?= Message-ID: <20180416214414.1.8E74E68E208C2728@mg.python.org> https://hg.python.org/jython/rev/623eaa3d7834 changeset: 8156:623eaa3d7834 user: Jeff Allen date: Mon Apr 16 20:56:52 2018 +0100 summary: Refactor PyJavaType.init for readability (no material change). files: src/org/python/core/PyJavaType.java | 686 +++++++++------ 1 files changed, 417 insertions(+), 269 deletions(-) diff --git a/src/org/python/core/PyJavaType.java b/src/org/python/core/PyJavaType.java --- a/src/org/python/core/PyJavaType.java +++ b/src/org/python/core/PyJavaType.java @@ -383,18 +383,25 @@ if (!Modifier.isPublic(forClass.getModifiers()) && !name.startsWith("org.python.core") && Options.respectJavaAccessibility) { handleSuperMethodArgCollisions(forClass); - return; + return; // XXX Why is it ok to skip the rest in this case? } - // Add methods and determine bean properties declared on this class - Map props = Generic.map(); - Map> events = Generic.map(); + /* + * Compile lists of the methods, fields and constructors to be exposed through this type. If + * we respect Java accessibility, this is simple. + */ Method[] methods; + Field[] fields; + Constructor[] constructors; + if (Options.respectJavaAccessibility) { - // returns just the public methods + // Just the public methods, fields and constructors methods = forClass.getMethods(); + fields = forClass.getFields(); + constructors = forClass.getConstructors(); + } else { - // Grab all methods on this class and all of its superclasses and make them accessible + // All methods on this class and all of its super classes List allMethods = Generic.list(); for (Class c = forClass; c != null; c = c.getSuperclass()) { for (Method meth : c.getDeclaredMethods()) { @@ -403,16 +410,129 @@ } } methods = allMethods.toArray(new Method[allMethods.size()]); + // Methods must be in resolution order. See issue #2391 for detail. + Arrays.sort(methods, new MethodComparator(new ClassComparator())); + + // All the fields on just this class + fields = forClass.getDeclaredFields(); + for (Field field : fields) { + field.setAccessible(true); + } + + // All the constructors (except if this is for class Class) + if (forClass == Class.class) { + // No matter the security manager, cannot set constructors accessible + constructors = forClass.getConstructors(); + } else { + constructors = forClass.getDeclaredConstructors(); + for (Constructor ctr : constructors) { + ctr.setAccessible(true); + } + } + } + + // Add methods, also accumulating them in reflectedFuncs, and spotting Java Bean members. + ArrayList reflectedFuncs = new ArrayList<>(methods.length); + Map props = Generic.map(); + Map> events = Generic.map(); + addMethods(baseClass, reflectedFuncs, props, events, methods); + addInheritedMethods(reflectedFuncs, methods); + + // Add fields declared on this type + addFields(baseClass, fields); + + // Fill in the bean events and properties picked up while going through the methods + addBeanEvents(events); + addBeanProperties(props); + + // Add constructors declared on this type + addConstructors(forClass, constructors); + + // Special handling for Java collection types + addCollectionProxies(forClass); + + // Handle classes that use the ClassDictInit pattern for their definition + PyObject nameSpecified = null; + if (ClassDictInit.class.isAssignableFrom(forClass) && forClass != ClassDictInit.class) { + try { + Method m = forClass.getMethod("classDictInit", PyObject.class); + m.invoke(null, dict); + // allow the class to override its name after it is loaded + nameSpecified = dict.__finditem__("__name__"); + if (nameSpecified != null) { + name = nameSpecified.toString(); + } + } catch (Exception exc) { + throw Py.JavaError(exc); + } } - // Make sure we sort all methods so they resolve in the right order. See #2391 for detail. - Arrays.sort(methods, new MethodComparator(new ClassComparator())); + // Fill in the __module__ attribute of PyReflectedFunctions. + if (reflectedFuncs.size() > 0) { + if (nameSpecified == null) { + nameSpecified = Py.newString(name); + } + for (PyReflectedFunction func : reflectedFuncs) { + func.__module__ = nameSpecified; + } + } + + // Handle descriptor classes + if (baseClass != Object.class) { + hasGet = getDescrMethod(forClass, "__get__", OO) != null + || getDescrMethod(forClass, "_doget", PyObject.class) != null + || getDescrMethod(forClass, "_doget", OO) != null; + hasSet = getDescrMethod(forClass, "__set__", OO) != null + || getDescrMethod(forClass, "_doset", OO) != null; + hasDelete = getDescrMethod(forClass, "__delete__", PyObject.class) != null + || getDescrMethod(forClass, "_dodel", PyObject.class) != null; + } + + /* + * Certain types get particular implementations of__lt__, __le__, __ge__, __gt__, __copy__ + * and __deepcopy__. + */ + if (forClass == Object.class) { + addMethodsForObject(); + } else if (forClass == Comparable.class) { + addMethodsForComparable(); + } else if (forClass == Cloneable.class) { + addMethodsForCloneable(); + } else if (forClass == Serializable.class) { + addMethodsForSerializable(); + } else if (immutableClasses.contains(forClass)) { + // __deepcopy__ just works for these objects since it uses serialization instead + addMethod(new PyBuiltinMethodNarrow("__copy__") { + + @Override + public PyObject __call__() { + return self; + } + }); + } + } + + /** + * Process the given class for methods defined on the target class itself (the + * fromClass), rather than inherited. + * + * This is exclusively a helper method for {@link #init(Set)}. + * + * @param baseClass ancestor of the target class + * @param reflectedFuncs to which reflected functions are added for further processing + * @param props to which Java Bean properties are added for further processing + * @param events to which Java Bean events are added for further processing + * @param methods of the target class + */ + private void addMethods(Class baseClass, List reflectedFuncs, + Map props, Map> events, + Method[] methods) { boolean isInAwt = name.startsWith("java.awt.") && name.indexOf('.', 9) == -1; - ArrayList reflectedFuncs = new ArrayList<>(methods.length); - PyReflectedFunction reflfunc; + + // First pass skip inherited (and certain "ignored") methods. for (Method meth : methods) { - if (!declaredOnMember(baseClass, meth) || ignore(meth)) { + if (!declaredHere(baseClass, meth) || ignore(meth)) { continue; } @@ -428,7 +548,8 @@ } String nmethname = normalize(methname); - reflfunc = (PyReflectedFunction) dict.__finditem__(nmethname); + + PyReflectedFunction reflfunc = (PyReflectedFunction) dict.__finditem__(nmethname); if (reflfunc == null) { reflfunc = new PyReflectedFunction(meth); reflectedFuncs.add(reflfunc); @@ -437,71 +558,100 @@ reflfunc.addMethod(meth); } - // Now check if this is a bean method, for which it must be an instance method - if (Modifier.isStatic(meth.getModifiers())) { - continue; - } + // Check if this is a Java Bean method, indicating the "bean nature" of the class + checkBeanMethod(props, events, meth); + } + } - // First check if this is a bean event addition method - int n = meth.getParameterTypes().length; - if ((methname.startsWith("add") || methname.startsWith("set")) - && methname.endsWith("Listener") && n == 1 && meth.getReturnType() == Void.TYPE - && EventListener.class.isAssignableFrom(meth.getParameterTypes()[0])) { - Class eventClass = meth.getParameterTypes()[0]; - String ename = eventClass.getName(); - int idot = ename.lastIndexOf('.'); - if (idot != -1) { - ename = ename.substring(idot + 1); - } - ename = normalize(StringUtil.decapitalize(ename)); - events.put(ename, new PyBeanEvent<>(ename, eventClass, meth)); - continue; - } + /** + * Consider whether the given method of the current class indicates the existence of a JavaBean + * property or event. + * + * @param props in which to store properties we discover + * @param events in which to store events we discover + * @param meth under consideration + */ + private void checkBeanMethod(Map props, + Map> events, Method meth) { + + // If this is a bean method at all, it must be an instance method + if (Modifier.isStatic(meth.getModifiers())) { + return; + } - // Now check if it's a bean property accessor - String beanPropertyName = null; - boolean get = true; - if (methname.startsWith("get") && methname.length() > 3 && n == 0) { - beanPropertyName = methname.substring(3); - } else if (methname.startsWith("is") && methname.length() > 2 && n == 0 - && meth.getReturnType() == Boolean.TYPE) { - beanPropertyName = methname.substring(2); - } else if (methname.startsWith("set") && methname.length() > 3 && n == 1) { - beanPropertyName = methname.substring(3); - get = false; + // First check if this is a bean event addition method + int n = meth.getParameterTypes().length; + String methname = meth.getName(); + if ((methname.startsWith("add") || methname.startsWith("set")) + && methname.endsWith("Listener") && n == 1 && meth.getReturnType() == Void.TYPE + && EventListener.class.isAssignableFrom(meth.getParameterTypes()[0])) { + // Yes, we have discovered an event type. Save the information for later. + Class eventClass = meth.getParameterTypes()[0]; + String ename = eventClass.getName(); + int idot = ename.lastIndexOf('.'); + if (idot != -1) { + ename = ename.substring(idot + 1); } - if (beanPropertyName != null) { - beanPropertyName = normalize(StringUtil.decapitalize(beanPropertyName)); - PyBeanProperty prop = props.get(beanPropertyName); - if (prop == null) { - prop = new PyBeanProperty(beanPropertyName, null, null, null); - props.put(beanPropertyName, prop); - } - if (get) { - prop.getMethod = meth; - prop.myType = meth.getReturnType(); - } else { - prop.setMethod = meth; - /* - * Needed for readonly properties. Getter will be used instead if there is one. - * Only works if setX method has exactly one param, which is the only reasonable - * case. - */ - // XXX: should we issue a warning if setX and getX have different types? - if (prop.myType == null) { - Class[] params = meth.getParameterTypes(); - if (params.length == 1) { - prop.myType = params[0]; - } + ename = normalize(StringUtil.decapitalize(ename)); + events.put(ename, new PyBeanEvent<>(ename, eventClass, meth)); + return; + } + + // Now check if it's a bean property accessor + String beanPropertyName = null; + boolean get = true; + if (methname.startsWith("get") && methname.length() > 3 && n == 0) { + beanPropertyName = methname.substring(3); + } else if (methname.startsWith("is") && methname.length() > 2 && n == 0 + && meth.getReturnType() == Boolean.TYPE) { + beanPropertyName = methname.substring(2); + } else if (methname.startsWith("set") && methname.length() > 3 && n == 1) { + beanPropertyName = methname.substring(3); + get = false; + } + + if (beanPropertyName != null) { + // Ok, its a bean property. Save this information for later. + beanPropertyName = normalize(StringUtil.decapitalize(beanPropertyName)); + PyBeanProperty prop = props.get(beanPropertyName); + if (prop == null) { + prop = new PyBeanProperty(beanPropertyName, null, null, null); + props.put(beanPropertyName, prop); + } + // getX and setX should be getting and setting the same type of thing. + if (get) { + prop.getMethod = meth; + prop.myType = meth.getReturnType(); + } else { + prop.setMethod = meth; + /* + * Needed for readonly properties. Getter will be used instead if there is one. Only + * works if setX method has exactly one param, which is the only reasonable case. + */ + // XXX: should we issue a warning if setX and getX have different types? + if (prop.myType == null) { + Class[] params = meth.getParameterTypes(); + if (params.length == 1) { + prop.myType = params[0]; } } } } + } - // Add superclass methods + /** + * Process the given class for methods inherited from ancestor classes. + * + * This is exclusively a helper method for {@link #init(Set)}. + * + * @param reflectedFuncs to which reflected functions are added for further processing + * @param methods of the target class + */ + private void addInheritedMethods(List reflectedFuncs, Method[] methods) { + // Add inherited and previously ignored methods for (Method meth : methods) { String nmethname = normalize(meth.getName()); - reflfunc = (PyReflectedFunction) dict.__finditem__(nmethname); + PyReflectedFunction reflfunc = (PyReflectedFunction) dict.__finditem__(nmethname); if (reflfunc != null) { /* * The superclass method has the same name as one declared on this class, so add the @@ -521,20 +671,20 @@ dict.__setitem__(nmethname, reflfunc); } } + } - // Add fields declared on this type - Field[] fields; - if (Options.respectJavaAccessibility) { - // returns just the public fields - fields = forClass.getFields(); - } else { - fields = forClass.getDeclaredFields(); - for (Field field : fields) { - field.setAccessible(true); - } - } + /** + * Process the given fields defined on the target class (the fromClass). + * + * This is exclusively a helper method for {@link #init(Set)}. + * + * + * @param baseClass + * @param fields + */ + private void addFields(Class baseClass, Field[] fields) { for (Field field : fields) { - if (!declaredOnMember(baseClass, field)) { + if (!declaredHere(baseClass, field)) { continue; } String fldname = field.getName(); @@ -559,7 +709,10 @@ dict.__setitem__(normalize(fldname), new PyReflectedField(field)); } } + } + /** Add the methods corresponding to a each discovered JavaBean event. */ + private void addBeanEvents(Map> events) { for (PyBeanEvent ev : events.values()) { if (dict.__finditem__(ev.__name__) == null) { dict.__setitem__(ev.__name__, ev); @@ -574,9 +727,13 @@ new PyBeanEventProperty(methodName, ev.eventClass, ev.addMethod, meth)); } } + } - // Fill in the bean properties picked up while going through the methods + /** Add the methods corresponding to a each discovered JavaBean property. */ + private void addBeanProperties(Map props) { for (PyBeanProperty prop : props.values()) { + + // Check for an existing __dict__ entry with this name PyObject prev = dict.__finditem__(prop.__name__); if (prev != null) { if (!(prev instanceof PyReflectedField) @@ -590,11 +747,12 @@ } /* - * If one of our superclasses has something defined for this name, check if its a bean + * If one of our super-classes has something defined for this name, check if it's a bean * property, and if so, try to fill in any gaps in our property from there. */ PyObject fromType[] = new PyObject[] {null}; PyObject superForName = lookup_where_mro(prop.__name__, fromType); + if (superForName instanceof PyBeanProperty) { PyBeanProperty superProp = ((PyBeanProperty) superForName); /* @@ -618,6 +776,7 @@ // If the parent bean is hiding a static field, we need it as well. prop.field = superProp.field; } + } else if (superForName != null && fromType[0] != this && !(superForName instanceof PyBeanEvent)) { /* @@ -627,6 +786,7 @@ */ continue; } + /* * If the return types on the set and get methods for a property don't agree, the get * method takes precedence. @@ -637,25 +797,23 @@ } dict.__setitem__(prop.__name__, prop); } + } + + /** + * Process the given constructors defined on the target class (the fromClass). + * + * This is exclusively a helper method for {@link #init(Set)}. + * + * @param forClass + * @param constructors + */ + private void addConstructors(Class forClass, Constructor[] constructors) { final PyReflectedConstructor reflctr = new PyReflectedConstructor(name); - Constructor[] constructors; - /* - * No matter the security manager, trying to set the constructor on class to accessible - * blows up. - */ - if (Options.respectJavaAccessibility || Class.class == forClass) { - // returns just the public constructors - constructors = forClass.getConstructors(); - } else { - constructors = forClass.getDeclaredConstructors(); - for (Constructor ctr : constructors) { - ctr.setAccessible(true); - } - } for (Constructor ctr : constructors) { reflctr.addConstructor(ctr); } + if (PyObject.class.isAssignableFrom(forClass)) { PyObject new_ = new PyNewWrapper(forClass, "__new__", -1, -1) { @@ -669,13 +827,23 @@ } else { dict.__setitem__("__init__", reflctr); } + } + + /** + * Add Python methods corresponding to the API of common Java collection types. This is + * exclusively a helper method for {@link #init(Set)}. + * + * @param forClass the target class + */ + private void addCollectionProxies(Class forClass) { PyBuiltinMethod[] collectionProxyMethods = getCollectionProxies().get(forClass); if (collectionProxyMethods != null) { for (PyBuiltinMethod meth : collectionProxyMethods) { addMethod(meth); } } - // allow for some methods to override the Java type's methods as a late injection + + // Allow for some methods to override the Java type's methods as a late injection for (Class type : getPostCollectionProxies().keySet()) { if (type.isAssignableFrom(forClass)) { for (PyBuiltinMethod meth : getPostCollectionProxies().get(type)) { @@ -683,191 +851,147 @@ } } } + } - PyObject nameSpecified = null; - if (ClassDictInit.class.isAssignableFrom(forClass) && forClass != ClassDictInit.class) { - try { - Method m = forClass.getMethod("classDictInit", PyObject.class); - m.invoke(null, dict); - // allow the class to override its name after it is loaded - nameSpecified = dict.__finditem__("__name__"); - if (nameSpecified != null) { - name = nameSpecified.toString(); - } - } catch (Exception exc) { - throw Py.JavaError(exc); + /** Add special methods when this PyJavaType represents Object. */ + private void addMethodsForObject() { + addMethod(new PyBuiltinMethodNarrow("__copy__") { + + @Override + public PyObject __call__() { + throw Py.TypeError( + "Could not copy Java object because it is not Cloneable or known to be immutable. " + + "Consider monkeypatching __copy__ for " + + self.getType().fastGetName()); } - } + }); + addMethod(new PyBuiltinMethodNarrow("__deepcopy__") { + + @Override + public PyObject __call__(PyObject memo) { + throw Py.TypeError("Could not deepcopy Java object because it is not Serializable. " + + "Consider monkeypatching __deepcopy__ for " + + self.getType().fastGetName()); + } + }); + addMethod(new PyBuiltinMethodNarrow("__eq__", 1) { - // Fill __module__ attribute of PyReflectedFunctions... - if (reflectedFuncs.size() > 0) { - if (nameSpecified == null) { - nameSpecified = Py.newString(name); + @Override + public PyObject __call__(PyObject o) { + Object proxy = self.getJavaProxy(); + Object oProxy = o.getJavaProxy(); + return proxy.equals(oProxy) ? Py.True : Py.False; } - for (PyReflectedFunction func : reflectedFuncs) { - func.__module__ = nameSpecified; - } - } + }); + addMethod(new PyBuiltinMethodNarrow("__ne__", 1) { - if (baseClass != Object.class) { - hasGet = getDescrMethod(forClass, "__get__", OO) != null - || getDescrMethod(forClass, "_doget", PyObject.class) != null - || getDescrMethod(forClass, "_doget", OO) != null; - hasSet = getDescrMethod(forClass, "__set__", OO) != null - || getDescrMethod(forClass, "_doset", OO) != null; - hasDelete = getDescrMethod(forClass, "__delete__", PyObject.class) != null - || getDescrMethod(forClass, "_dodel", PyObject.class) != null; - } + @Override + public PyObject __call__(PyObject o) { + Object proxy = self.getJavaProxy(); + Object oProxy = o.getJavaProxy(); + return !proxy.equals(oProxy) ? Py.True : Py.False; + } + }); + addMethod(new PyBuiltinMethodNarrow("__hash__") { - if (forClass == Object.class) { - addMethod(new PyBuiltinMethodNarrow("__copy__") { + @Override + public PyObject __call__() { + return Py.newInteger(self.getJavaProxy().hashCode()); + } + }); + addMethod(new PyBuiltinMethodNarrow("__repr__") { - @Override - public PyObject __call__() { - throw Py.TypeError( - "Could not copy Java object because it is not Cloneable or known to be immutable. " - + "Consider monkeypatching __copy__ for " - + self.getType().fastGetName()); - } - }); - addMethod(new PyBuiltinMethodNarrow("__deepcopy__") { + @Override + public PyObject __call__() { + /* + * java.lang.Object.toString returns Unicode: preserve as a PyUnicode, then let the + * repr() built-in decide how to handle it. (Also applies to __str__.) + */ + String toString = self.getJavaProxy().toString(); + return toString == null ? Py.EmptyUnicode : Py.newUnicode(toString); + } + }); + addMethod(new PyBuiltinMethodNarrow("__unicode__") { - @Override - public PyObject __call__(PyObject memo) { - throw Py.TypeError( - "Could not deepcopy Java object because it is not Serializable. " - + "Consider monkeypatching __deepcopy__ for " - + self.getType().fastGetName()); - } - }); - addMethod(new PyBuiltinMethodNarrow("__eq__", 1) { + @Override + public PyObject __call__() { + return new PyUnicode(self.toString()); + } + }); + } + + /** Add special methods when this PyJavaType represents interface Comparable. */ + private void addMethodsForComparable() { + addMethod(new ComparableMethod("__lt__", 1) { + + @Override + protected boolean getResult(int comparison) { + return comparison < 0; + } + }); + addMethod(new ComparableMethod("__le__", 1) { - @Override - public PyObject __call__(PyObject o) { - Object proxy = self.getJavaProxy(); - Object oProxy = o.getJavaProxy(); - return proxy.equals(oProxy) ? Py.True : Py.False; - } - }); - addMethod(new PyBuiltinMethodNarrow("__ne__", 1) { + @Override + protected boolean getResult(int comparison) { + return comparison <= 0; + } + }); + addMethod(new ComparableMethod("__gt__", 1) { + + @Override + protected boolean getResult(int comparison) { + return comparison > 0; + } + }); + addMethod(new ComparableMethod("__ge__", 1) { + + @Override + protected boolean getResult(int comparison) { + return comparison >= 0; + } + }); + } - @Override - public PyObject __call__(PyObject o) { - Object proxy = self.getJavaProxy(); - Object oProxy = o.getJavaProxy(); - return !proxy.equals(oProxy) ? Py.True : Py.False; - } - }); - addMethod(new PyBuiltinMethodNarrow("__hash__") { + /** Add special methods when this PyJavaType represents interface Cloneable. */ + private void addMethodsForCloneable() { + addMethod(new PyBuiltinMethodNarrow("__copy__") { - @Override - public PyObject __call__() { - return Py.newInteger(self.getJavaProxy().hashCode()); - } - }); - addMethod(new PyBuiltinMethodNarrow("__repr__") { - - @Override - public PyObject __call__() { - /* - * java.lang.Object.toString returns Unicode: preserve as a PyUnicode, then let - * the repr() built-in decide how to handle it. (Also applies to __str__.) - */ - String toString = self.getJavaProxy().toString(); - return toString == null ? Py.EmptyUnicode : Py.newUnicode(toString); + @Override + public PyObject __call__() { + Object obj = self.getJavaProxy(); + Method clone; + /* + * TODO we could specialize so that for well known objects like collections. This + * would avoid needing to use reflection in the general case, because Object#clone + * is protected (but most subclasses are not). Lastly we can potentially cache the + * method handle in the proxy instead of looking it up each time + */ + try { + clone = obj.getClass().getMethod("clone"); + Object copy = clone.invoke(obj); + return Py.java2py(copy); + } catch (Exception ex) { + throw Py.TypeError("Could not copy Java object"); } - }); - addMethod(new PyBuiltinMethodNarrow("__unicode__") { - - @Override - public PyObject __call__() { - return new PyUnicode(self.toString()); - } - }); - } - - if (forClass == Comparable.class) { - addMethod(new ComparableMethod("__lt__", 1) { - - @Override - protected boolean getResult(int comparison) { - return comparison < 0; - } - }); - addMethod(new ComparableMethod("__le__", 1) { + } + }); + } - @Override - protected boolean getResult(int comparison) { - return comparison <= 0; - } - }); - addMethod(new ComparableMethod("__gt__", 1) { - - @Override - protected boolean getResult(int comparison) { - return comparison > 0; - } - }); - addMethod(new ComparableMethod("__ge__", 1) { - - @Override - protected boolean getResult(int comparison) { - return comparison >= 0; - } - }); - } - - if (immutableClasses.contains(forClass)) { - // __deepcopy__ just works for these objects since it uses serialization instead - addMethod(new PyBuiltinMethodNarrow("__copy__") { + /** Add special methods when this PyJavaType represents interface Serializable. */ + private void addMethodsForSerializable() { + addMethod(new PyBuiltinMethodNarrow("__deepcopy__") { - @Override - public PyObject __call__() { - return self; + @Override + public PyObject __call__(PyObject memo) { + Object obj = self.getJavaProxy(); + try { + Object copy = cloneX(obj); + return Py.java2py(copy); + } catch (Exception ex) { + throw Py.TypeError("Could not copy Java object"); } - }); - } - - if (forClass == Cloneable.class) { - addMethod(new PyBuiltinMethodNarrow("__copy__") { - - @Override - public PyObject __call__() { - Object obj = self.getJavaProxy(); - Method clone; - /* - * TODO we could specialize so that for well known objects like collections. - * This would avoid needing to use reflection in the general case, because - * Object#clone is protected (but most subclasses are not). Lastly we can - * potentially cache the method handle in the proxy instead of looking it up - * each time - */ - try { - clone = obj.getClass().getMethod("clone"); - Object copy = clone.invoke(obj); - return Py.java2py(copy); - } catch (Exception ex) { - throw Py.TypeError("Could not copy Java object"); - } - } - }); - } - - if (forClass == Serializable.class) { - addMethod(new PyBuiltinMethodNarrow("__deepcopy__") { - - @Override - public PyObject __call__(PyObject memo) { - Object obj = self.getJavaProxy(); - try { - Object copy = cloneX(obj); - return Py.java2py(copy); - } catch (Exception ex) { - throw Py.TypeError("Could not copy Java object"); - } - } - }); - } + } + }); } /* @@ -969,6 +1093,14 @@ } } + /** + * From a given class that is an ancestor of the Java class for which this PyJavaType is being + * initialised, or that is an interface implemented by it or an ancestor, process each method in + * the ancestor so that, if the class or interface that declares the method is public, that + * method becomes a method of the Python type we are constructing. + * + * @param parent class or interface + */ private void mergeMethods(Class parent) { for (Method meth : parent.getMethods()) { if (!Modifier.isPublic(meth.getDeclaringClass().getModifiers())) { @@ -1004,9 +1136,24 @@ } } - private static boolean declaredOnMember(Class base, Member declaring) { - return base == null || (declaring.getDeclaringClass() != base - && base.isAssignableFrom(declaring.getDeclaringClass())); + /** + * True iff the target of this PyJavaType's target (the forClass) + * declares the given member of this target. For this, the method is supplied the super-class of + * the target. + * + * @param baseClass super-class of the target of this PyJavaType, or + * null. + * @param member of the forClass that might be exposed on this + * PyJavaType + * @return true if the member is declared here on the fromClass + */ + private static boolean declaredHere(Class baseClass, Member member) { + if (baseClass == null) { + return true; + } else { + Class declaring = member.getDeclaringClass(); + return declaring != baseClass && baseClass.isAssignableFrom(declaring); + } } private static String normalize(String name) { @@ -1029,6 +1176,7 @@ return null; } + /** Recognise certain methods as ignored (tagged as throws PyIgnoreMethodTag). */ private static boolean ignore(Method meth) { Class[] exceptions = meth.getExceptionTypes(); for (Class exception : exceptions) { -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Thu Apr 19 17:25:51 2018 From: jython-checkins at python.org (jeff.allen) Date: Thu, 19 Apr 2018 21:25:51 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Restore_ordering_of_method?= =?utf-8?q?_list_in_PyJavaType=2E?= Message-ID: <20180419212551.1.9B9C76B0F8EBE341@mg.python.org> https://hg.python.org/jython/rev/2adcf0e29407 changeset: 8157:2adcf0e29407 user: Jeff Allen date: Thu Apr 19 22:16:21 2018 +0100 summary: Restore ordering of method list in PyJavaType. Accidentally displaced in previous refactoring. (Oddly, Issue2391AttrOrderTest passed.) files: src/org/python/core/PyJavaType.java | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/org/python/core/PyJavaType.java b/src/org/python/core/PyJavaType.java --- a/src/org/python/core/PyJavaType.java +++ b/src/org/python/core/PyJavaType.java @@ -410,8 +410,6 @@ } } methods = allMethods.toArray(new Method[allMethods.size()]); - // Methods must be in resolution order. See issue #2391 for detail. - Arrays.sort(methods, new MethodComparator(new ClassComparator())); // All the fields on just this class fields = forClass.getDeclaredFields(); @@ -430,6 +428,8 @@ } } } + // Methods must be in resolution order. See issue #2391 for detail. + Arrays.sort(methods, new MethodComparator(new ClassComparator())); // Add methods, also accumulating them in reflectedFuncs, and spotting Java Bean members. ArrayList reflectedFuncs = new ArrayList<>(methods.length); -- Repository URL: https://hg.python.org/jython From jython-checkins at python.org Mon Apr 30 04:06:49 2018 From: jython-checkins at python.org (jeff.allen) Date: Mon, 30 Apr 2018 08:06:49 +0000 Subject: [Jython-checkins] =?utf-8?q?jython=3A_Avoid_reflective_access_wa?= =?utf-8?q?rnings_on_modular_Java_platform=2E_Fixes_=232662=2E?= Message-ID: <20180430080649.1.61329D5AE9ED0732@mg.python.org> https://hg.python.org/jython/rev/78b574d6a6f7 changeset: 8158:78b574d6a6f7 user: Jeff Allen date: Mon Apr 30 07:58:24 2018 +0100 summary: Avoid reflective access warnings on modular Java platform. Fixes #2662. This change extends the applicability of handleSuperMethodArgCollisions (in PyJavaType) to cover classes inaccessible as a result of modular platform access controls introduced in Java 9 (project Jigsaw). The apparatus for this test is obtained reflectively at run-time, since it doesn't exist in Java 7 and 8. files: NEWS | 1 + src/org/python/core/PyJavaType.java | 131 +++++++++++++-- 2 files changed, 109 insertions(+), 23 deletions(-) diff --git a/NEWS b/NEWS --- a/NEWS +++ b/NEWS @@ -4,6 +4,7 @@ Developement tip Bugs fixed + - [ 2662 ] IllegalAccessException accessing public abstract method via PyReflectedFunction - [ 2501 ] JAVA_STACK doesn't work (fixed for Windows launcher only) - [ 1866 ] Parser does not have mismatch token error messages caught by BaseRecognizer - [ 1930 ] traceback raises exception in os.py diff --git a/src/org/python/core/PyJavaType.java b/src/org/python/core/PyJavaType.java --- a/src/org/python/core/PyJavaType.java +++ b/src/org/python/core/PyJavaType.java @@ -10,6 +10,10 @@ import java.io.ObjectStreamClass; import java.io.OutputStream; import java.io.Serializable; +import java.lang.invoke.MethodHandle; +import java.lang.invoke.MethodHandles; +import java.lang.invoke.MethodHandles.Lookup; +import java.lang.invoke.MethodType; import java.lang.reflect.Constructor; import java.lang.reflect.Field; import java.lang.reflect.Member; @@ -380,10 +384,9 @@ * PyReflected* can't call or access anything from non-public classes that aren't in * org.python.core */ - if (!Modifier.isPublic(forClass.getModifiers()) && !name.startsWith("org.python.core") - && Options.respectJavaAccessibility) { + if (!isAccessibleClass(forClass) && !name.startsWith("org.python.core")) { handleSuperMethodArgCollisions(forClass); - return; // XXX Why is it ok to skip the rest in this case? + return; } /* @@ -513,6 +516,81 @@ } /** + * A class containing a test that a given class is accessible to Jython, in the Modular Java + * sense. It is in its own class simply to contain the state we need and its messy + * initialisation. + */ + private static class Modular { + + private static final Lookup LOOKUP = MethodHandles.lookup(); + + /** + * Test we need is constructed as a method handle, reflectively (so it works on Java less + * than 9). + */ + private static final MethodHandle accessibleMH; + static { + MethodHandle acc; + try { + Class moduleClass = Class.forName("java.lang.Module"); + // mod = ?(c) c.getModule() : Class ? Module + MethodHandle mod = LOOKUP.findVirtual(Class.class, "getModule", + MethodType.methodType(moduleClass)); + // pkg = ?(c) c.getPackageName() : Class ? String + MethodHandle pkg = LOOKUP.findVirtual(Class.class, "getPackageName", + MethodType.methodType(String.class)); + // exps = ?(m, pn) m.isExported(pn) : Module, String ? boolean + MethodHandle exps = LOOKUP.findVirtual(moduleClass, "isExported", + MethodType.methodType(boolean.class, String.class)); + // expc = ?(m, c) exps(m, pkg(c)) : Module, Class ? boolean + MethodHandle expc = MethodHandles.filterArguments(exps, 1, pkg); + // acc = ?(c) expc(mod(c), c) : Class ? boolean + acc = MethodHandles.foldArguments(expc, mod); + } catch (ReflectiveOperationException | SecurityException e) { + // Assume not a modular Java platform: acc = ?(c) true : Class ? boolean + acc = MethodHandles.dropArguments(MethodHandles.constant(boolean.class, true), 0, + Class.class); + } + accessibleMH = acc; + } + + /** + * Test whether a given class is in a package exported (accessible) to Jython, in the + * Modular Java sense. Jython code is all in the unnamed module, so in fact we are testing + * accessibility to the package of the class. + * + * If this means nothing on this platform (if the classes and methods needed are not found), + * decide that we are not on a modular platform, in which case all invocations return + * true. + * + * @param c the class + * @return true iff accessible to Jython + */ + static boolean accessible(Class c) { + try { + return (boolean) accessibleMH.invokeExact(c); + } catch (Throwable e) { + return true; + } + } + } + + /** + * Test whether a given class is accessible, meaning it is in a package exported to Jython and + * public (or we are ignoring accessibility). + * + * @param c the class + * @return true iff accessible to Jython + */ + static boolean isAccessibleClass(Class c) { + if (!Modular.accessible(c)) { + return false; + } else { + return !Options.respectJavaAccessibility || Modifier.isPublic(c.getModifiers()); + } + } + + /** * Process the given class for methods defined on the target class itself (the * fromClass), rather than inherited. * @@ -1063,32 +1141,39 @@ } /** - * Private, protected or package protected classes that implement public interfaces or extend - * public classes can't have their implementations of the methods of their supertypes called - * through reflection due to Sun VM bug 4071957(http://tinyurl.com/le9vo). They can be called - * through the supertype version of the method though. Unfortunately we can't just let normal - * mro lookup of those methods handle routing the call to the correct version as a class can - * implement interfaces or classes that each have methods with the same name that takes - * different number or types of arguments. Instead this method goes through all interfaces - * implemented by this class, and combines same-named methods into a single PyReflectedFunction. + * Methods implemented in private, protected or package protected classes, and classes not + * exported by their module, may not be called directly through reflection. This is discussed in + * JDK issue 4283544, + * and is not likely to change. They may however be called through the Method object of the + * accessible class or interface that they implement. In non-reflective code, the Java compiler + * would generate such a call, based on the type declared for the target, which must + * therefore be accessible. + *

+ * An MRO lookup on the actual class will find a PyReflectedFunction that defines + * the method to Python. The normal process for creating that object will add the + * inaccessible implementation method(s) to the list, not those of the public API. (A + * class can of course have several methods with the same name and different signatures and the + * PyReflectedFunction lists them all.) We must therefore take care to enter the + * corresponding accessible API methods instead. + *

+ * Prior to Jython 2.5, this was handled by setting methods in package protected classes + * accessible which made them callable through reflection. That had the drawback of failing when + * running in a security environment that didn't allow setting accessibility, and will fail on + * the modular Java platform, so this method replaced it. * - * Prior to Jython 2.5, this was handled in PyJavaClass.setMethods by setting methods in package - * protected classes accessible which made them callable through reflection. That had the - * drawback of failing when running in a security environment that didn't allow setting - * accessibility, so this method replaced it. + * @param forClass of which the methods are currently being defined */ private void handleSuperMethodArgCollisions(Class forClass) { for (Class iface : forClass.getInterfaces()) { mergeMethods(iface); } - if (forClass.getSuperclass() != null) { - mergeMethods(forClass.getSuperclass()); - if (!Modifier.isPublic(forClass.getSuperclass().getModifiers())) { - /* - * If the superclass is also not public, it needs to get the same treatment as we - * can't call its methods either. - */ - handleSuperMethodArgCollisions(forClass.getSuperclass()); + Class parent = forClass.getSuperclass(); + if (parent != null) { + if (isAccessibleClass(parent)) { + mergeMethods(parent); + } else { + // The parent class is also not public: go up one more in the ancestry. + handleSuperMethodArgCollisions(parent); } } } -- Repository URL: https://hg.python.org/jython