From jython-checkins at python.org  Mon Mar  2 02:11:53 2015
From: jython-checkins at python.org (jim.baker)
Date: Mon, 02 Mar 2015 01:11:53 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Add_java=5Fuser_scheme_to_s?=
 =?utf-8?q?upport_user_directory_installs?=
Message-ID: <20150302011152.77536.25201@psf.io>

https://hg.python.org/jython/rev/c32815d44800
changeset:   7599:c32815d44800
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sun Mar 01 18:11:52 2015 -0700
summary:
  Add java_user scheme to support user directory installs

Fixes http://bugs.jython.org/issue2189

files:
  Lib/distutils/command/install.py |  9 ++++++++-
  1 files changed, 8 insertions(+), 1 deletions(-)


diff --git a/Lib/distutils/command/install.py b/Lib/distutils/command/install.py
--- a/Lib/distutils/command/install.py
+++ b/Lib/distutils/command/install.py
@@ -89,7 +89,14 @@
         'headers': '$base/Include/$dist_name',
         'scripts': '$base/bin',
         'data'   : '$base',
-        }
+        },
+    'java_user': {
+        'purelib': '$usersite',
+        'platlib': '$usersite',
+        'headers': '$userbase/include/python$py_version_short/$dist_name',
+        'scripts': '$userbase/bin',
+        'data'   : '$userbase',
+        },
     }
 
 # The keys to an installation scheme; if any new types of files are to be

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Mon Mar  2 02:26:35 2015
From: jython-checkins at python.org (jim.baker)
Date: Mon, 02 Mar 2015 01:26:35 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Catch_RuntimeException_in_h?=
 =?utf-8?q?andling_sun=2Emisc=2ESignal_exceptions?=
Message-ID: <20150302012634.10080.86023@psf.io>

https://hg.python.org/jython/rev/75513a17baf9
changeset:   7600:75513a17baf9
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sun Mar 01 18:26:09 2015 -0700
summary:
  Catch RuntimeException in handling sun.misc.Signal exceptions

Use RuntimeException, the super class of
java.lang.IllegalArgumentException, on systems that wrap the returned
exceptions with RuntimeException, when working with signals.

Fixes http://bugs.jython.org/issue2084

files:
  Lib/signal.py |  14 ++++++++------
  1 files changed, 8 insertions(+), 6 deletions(-)


diff --git a/Lib/signal.py b/Lib/signal.py
--- a/Lib/signal.py
+++ b/Lib/signal.py
@@ -40,14 +40,16 @@
 import sys
 import threading
 import time
-from java.lang import IllegalArgumentException
+from java.lang import RuntimeException
 from java.util.concurrent.atomic import AtomicReference
 
 debug = 0
 
 def _init_signals():
-    # install signals by checking for standard names
-    # using IllegalArgumentException to diagnose
+    # Install signals by checking for standard names, by using
+    # RuntimeException to diagnose. On some platforms like J9,
+    # RuntimeException wraps IllegalArgumentException, but it's also
+    # the superclass of IllegalArgumentException.
 
     possible_signals = """
         SIGABRT
@@ -89,7 +91,7 @@
     for signal_name in possible_signals:
         try:
             java_signal = sun.misc.Signal(signal_name[3:])
-        except IllegalArgumentException:
+        except RuntimeException:
             continue
 
         signal_number = java_signal.getNumber()
@@ -147,7 +149,7 @@
 def _register_signal(signal, action):
     try:
         return sun.misc.Signal.handle(signal, action)
-    except IllegalArgumentException, err:
+    except RuntimeException, err:
         raise ValueError(err.getMessage())
 
 
@@ -231,7 +233,7 @@
     def raise_alarm():
         try:
             sun.misc.Signal.raise(_signals[SIGALRM])
-        except IllegalArgumentException, err:
+        except RuntimeException, err:
             raise ValueError(err.getMessage())
 
     if time > 0:

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Mon Mar  2 02:29:36 2015
From: jython-checkins at python.org (jim.baker)
Date: Mon, 02 Mar 2015 01:29:36 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Print_retrying_for_test=5Fs?=
 =?utf-8?q?upport_retry_decorator_only_if_verbose?=
Message-ID: <20150302012850.77544.65508@psf.io>

https://hg.python.org/jython/rev/5d7c7759eee9
changeset:   7601:5d7c7759eee9
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sun Mar 01 18:28:45 2015 -0700
summary:
  Print retrying for test_support retry decorator only if verbose

files:
  Lib/test/test_support.py |  3 ++-
  1 files changed, 2 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py
--- a/Lib/test/test_support.py
+++ b/Lib/test/test_support.py
@@ -1456,7 +1456,8 @@
                 try:
                     return f(*args, **kwds)
                 except exceptions as e:
-                    print "Got %s, retrying in %.2f seconds..." % (str(e), mdelay)
+                    if verbose:
+                        print "Got %s, retrying in %.2f seconds..." % (str(e), mdelay)
                     # FIXME resource cleanup continues to be an issue
                     # in terms of tests we use from CPython. This only
                     # represents a bandaid - useful as it might be -

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Tue Mar  3 06:24:52 2015
From: jython-checkins at python.org (jim.baker)
Date: Tue, 03 Mar 2015 05:24:52 +0000
Subject: [Jython-checkins] =?utf-8?b?anl0aG9uOiBSZXN0b3JlcyBfX3RvamF2YV9f?=
 =?utf-8?q?_to_datetime=2E=7Bdate=2C_datetime=2C_time=7D_classes?=
Message-ID: <20150303052438.3291.54160@psf.io>

https://hg.python.org/jython/rev/67210f355576
changeset:   7602:67210f355576
user:        Jim Baker <jim.baker at rackspace.com>
date:        Mon Mar 02 22:24:31 2015 -0700
summary:
  Restores __tojava__ to datetime.{date, datetime, time} classes

Also adds timezone support compared to original implementation. Tests
conversions in test_datetime_jy, and this test now runs in regrtest.

Fixes http://bugs.jython.org/issue2271

files:
  Lib/datetime.py              |   88 ++++++++++++++++++
  Lib/test/test_datetime_jy.py |  111 +++++++++++++++++++++-
  2 files changed, 192 insertions(+), 7 deletions(-)


diff --git a/Lib/datetime.py b/Lib/datetime.py
--- a/Lib/datetime.py
+++ b/Lib/datetime.py
@@ -20,6 +20,49 @@
 import time as _time
 import math as _math
 import struct as _struct
+import sys as _sys
+
+if _sys.platform.startswith('java'):
+    from java.lang import Object
+    from java.sql import Date, Time, Timestamp
+    from java.util import Calendar, GregorianCalendar, TimeZone
+    from org.python.core import Py
+
+    # Java does have the distinction between naive and aware times and
+    # datetimes provided by Python. To avoid using the local timezone,
+    # we assume it is UTC as this is most conventional (and least
+    # surprising). See for example - or in this library itself! -
+    # https://docs.python.org/2.7/library/datetime.html#datetime.datetime.utcnow
+    # - tzinfo is None
+
+    _utc_timezone = TimeZone.getTimeZone("UTC")
+    _is_jython = True
+
+    def _make_java_utc_calendar():
+        cal = GregorianCalendar(_utc_timezone)
+        cal.clear()
+        return cal
+
+    def _make_java_calendar(d):
+        tzinfo = d.tzinfo
+        if tzinfo == None:
+            cal = GregorianCalendar(_utc_timezone)
+        else:
+            tzname = tzinfo.tzname(d)
+            if tzname != None:
+                # Only applicable if Python code is using the Olson
+                # timezone database, which is what Java uses
+                tz = TimeZone.getTimeZone(tzname)
+                if tz.getID() != tzname and tz.getID() == "GMT":
+                    cal = GregorianCalendar(_utc_timezone)
+                    cal.set(Calendar.DST_OFFSET, int(tzinfo.dst(d).total_seconds() * 1000))
+                    cal.set(Calendar.ZONE_OFFSET, int(tzinfo.utcoffset(d).total_seconds() * 1000))
+                else:
+                    cal = GregorianCalendar(tz)
+        return cal
+else:
+    _is_jython = False
+
 
 def _cmp(x, y):
     return 0 if x == y else 1 if x > y else -1
@@ -1026,6 +1069,18 @@
     def __reduce__(self):
         return (self.__class__, self._getstate())
 
+    if _is_jython:
+        def __tojava__(self, java_class):
+            if java_class not in (Calendar, Date, Object):
+                return Py.NoConversion
+            calendar = _make_java_utc_calendar()
+            calendar.set(self.year, self.month - 1, self.day)
+            if java_class == Calendar:
+                return calendar
+            else:
+                return Date(calendar.getTimeInMillis())
+
+
 _date_class = date  # so functions w/ args named "date" can get at the class
 
 date.min = date(1, 1, 1)
@@ -1431,6 +1486,21 @@
     def __reduce__(self):
         return (time, self._getstate())
 
+    if _is_jython:
+        def __tojava__(self, java_class):
+            if java_class not in (Calendar, Time, Object):
+                return Py.NoConversion
+            calendar = _make_java_calendar(self)
+            if calendar == Py.NoConversion:
+                return Py.NoConversion
+            epoch_ms = (self.hour * 3600 + self.minute * 60 + self.second) * 1000 + self.microsecond // 1000
+            calendar.setTimeInMillis(epoch_ms)
+            if java_class == Calendar:
+                return calendar
+            else:
+                return Time(calendar.getTimeInMillis())
+
+
 _time_class = time  # so functions w/ args named "time" can get at the class
 
 time.min = time(0, 0, 0)
@@ -1928,6 +1998,24 @@
     def __reduce__(self):
         return (self.__class__, self._getstate())
 
+    if _is_jython:
+        def __tojava__(self, java_class):
+            if java_class not in (Calendar, Timestamp, Object):
+                return Py.NoConversion
+            calendar = _make_java_calendar(self)
+            if calendar == Py.NoConversion:
+                return Py.NoConversion
+            calendar.set(self.year, self.month - 1, self.day,
+                         self.hour, self.minute, self.second)
+
+            if java_class == Calendar:
+                calendar.set(Calendar.MILLISECOND, self.microsecond // 1000)
+                return calendar
+            else:
+                timestamp = Timestamp(calendar.getTimeInMillis())
+                timestamp.setNanos(self.microsecond * 1000)
+                return timestamp
+
 
 datetime.min = datetime(1, 1, 1)
 datetime.max = datetime(9999, 12, 31, 23, 59, 59, 999999)
diff --git a/Lib/test/test_datetime_jy.py b/Lib/test/test_datetime_jy.py
--- a/Lib/test/test_datetime_jy.py
+++ b/Lib/test/test_datetime_jy.py
@@ -6,30 +6,127 @@
 from datetime import time
 from datetime import date, datetime
 
-from java.util import Calendar
-from java.sql import Date
+from java.util import Calendar, GregorianCalendar
+from java.sql import Date, Time, Timestamp
 
 
-class TestJavaDatetime(unittest.TestCase):
+class TestCalendar(unittest.TestCase):
 
     def test_datetime(self):
         self.assertTrue(hasattr(datetime, "__tojava__"))
         x = datetime(2007, 1, 3)
         y = x.__tojava__(Calendar)
-        self.assertTrue(isinstance(y, Calendar))
+        self.assertIsInstance(y, GregorianCalendar)
+        self.assertEqual(y.get(Calendar.YEAR), 2007)
+        self.assertEqual(y.get(Calendar.MONTH), 0)
+        self.assertEqual(y.get(Calendar.DAY_OF_MONTH), 3)
+        self.assertEqual(y.get(Calendar.HOUR), 0)
+        self.assertEqual(y.get(Calendar.MINUTE), 0)
+        self.assertEqual(y.get(Calendar.SECOND), 0)
 
     def test_date(self):
         self.assertTrue(hasattr(date, "__tojava__"))
         x = date(2007, 1, 3)
         y = x.__tojava__(Calendar)
-        self.assertTrue(isinstance(y, Calendar))
+        self.assertIsInstance(y, GregorianCalendar)
+        self.assertEqual(y.get(Calendar.YEAR), 2007)
+        self.assertEqual(y.get(Calendar.MONTH), 0)
+        self.assertEqual(y.get(Calendar.DAY_OF_MONTH), 3)
 
     def test_time(self):
         self.assertTrue(hasattr(time, "__tojava__"))
         x = time(1, 3)
         y = x.__tojava__(Calendar)
-        self.assertTrue(isinstance(y, Calendar))
+        self.assertIsInstance(y, GregorianCalendar)
+        # Note obvious implementation details from GregorianCalendar
+        # and its definition in terms of the epoch
+        self.assertEqual(y.get(Calendar.YEAR), 1970)
+        self.assertEqual(y.get(Calendar.MONTH), 0)
+        self.assertEqual(y.get(Calendar.DAY_OF_MONTH), 1)
+        self.assertEqual(y.get(Calendar.HOUR), 1)
+        self.assertEqual(y.get(Calendar.MINUTE), 3)
+        self.assertEqual(y.get(Calendar.SECOND), 0)
+
+
+class TestTimezone(unittest.TestCase):
+
+    def test_olson(self):
+        class GMT1(tzinfo):
+            def utcoffset(self, dt):
+                return timedelta(hours=1)
+            def dst(self, dt):
+                return timedelta(0)
+            def tzname(self,dt):
+                return "Europe/Prague"
+
+        self.assertTrue(hasattr(datetime, "__tojava__"))
+        x = datetime(2007, 1, 3, tzinfo=GMT1())
+        y = x.__tojava__(Calendar)
+        self.assertIsInstance(y, GregorianCalendar)
+        self.assertEqual(y.get(Calendar.YEAR), 2007)
+        self.assertEqual(y.get(Calendar.MONTH), 0)
+        self.assertEqual(y.get(Calendar.DAY_OF_MONTH), 3)
+        self.assertEqual(y.get(Calendar.HOUR), 0)
+        self.assertEqual(y.get(Calendar.MINUTE), 0)
+        self.assertEqual(y.get(Calendar.SECOND), 0)
+        self.assertEqual(y.getTimeZone().getID(), "Europe/Prague")
+
+    def test_offset(self):
+        class Offset(tzinfo):
+            def utcoffset(self, dt):
+                return timedelta(hours=1, minutes=15)
+            def dst(self, dt):
+                return timedelta(seconds=-900)
+            def tzname(self,dt):
+                return "Foo/Bar"
+
+        self.assertTrue(hasattr(datetime, "__tojava__"))
+        x = datetime(2007, 1, 3, tzinfo=Offset())
+        y = x.__tojava__(Calendar)
+        self.assertIsInstance(y, GregorianCalendar)
+        self.assertEqual(y.get(Calendar.YEAR), 2007)
+        self.assertEqual(y.get(Calendar.MONTH), 0)
+        self.assertEqual(y.get(Calendar.DAY_OF_MONTH), 3)
+        self.assertEqual(y.get(Calendar.HOUR), 0)
+        self.assertEqual(y.get(Calendar.MINUTE), 0)
+        self.assertEqual(y.get(Calendar.SECOND), 0)
+        self.assertEqual(y.getTimeZone().getID(), "UTC")
+        self.assertEqual(y.get(Calendar.DST_OFFSET), -900 * 1000)
+        self.assertEqual(y.get(Calendar.ZONE_OFFSET), 4500 * 1000)
+
+
+class TestSQL(unittest.TestCase):
+
+    def test_datetime(self):
+        self.assertTrue(hasattr(datetime, "__tojava__"))
+        x = datetime(2007, 1, 3)
+        y = x.__tojava__(Timestamp)
+        self.assertIsInstance(y, Timestamp)
+        self.assertEqual(y.getTime(), (x - datetime(1970, 1, 1)).total_seconds() * 1000)
+
+    def test_date(self):
+        self.assertTrue(hasattr(date, "__tojava__"))
+        x = date(2007, 1, 3)
+        y = x.__tojava__(Date)
+        self.assertIsInstance(y, Date)
+        self.assertEqual(y.getTime(), (x - date(1970, 1, 1)).total_seconds() * 1000)
+
+    def test_time(self):
+        self.assertTrue(hasattr(time, "__tojava__"))
+        x = time(1, 3)
+        y = x.__tojava__(Time)
+        self.assertIsInstance(y, Time)
+        epoch = y.getTime()/1000.
+        self.assertEqual(epoch // 3600, 1)   # 1 hour
+        self.assertEqual(epoch % 3600, 180)  # 3 minutes
+
+
+def test_main():
+    test_support.run_unittest(
+        TestCalendar,
+        TestSQL,
+        TestTimezone)
 
 
 if __name__ == '__main__':
-    unittest.main()
+    test_main()

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar  7 15:45:53 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 07 Mar 2015 14:45:53 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_time=2Esleep=280=29_should_?=
	=?utf-8?q?always_attempt_to_yield?=
Message-ID: <20150307144552.1355.46581@psf.io>

https://hg.python.org/jython/rev/d404d911f3e9
changeset:   7603:d404d911f3e9
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 06 23:55:26 2015 -0700
summary:
  time.sleep(0) should always attempt to yield

Use java.lang.Thread.yield() instead of j.l.Thread.sleep when the
sleep time is zero, to conform better with CPython's scheduling
semantics.

files:
  src/org/python/modules/time/Time.java |  19 ++++++++++----
  1 files changed, 13 insertions(+), 6 deletions(-)


diff --git a/src/org/python/modules/time/Time.java b/src/org/python/modules/time/Time.java
--- a/src/org/python/modules/time/Time.java
+++ b/src/org/python/modules/time/Time.java
@@ -449,12 +449,19 @@
     }
 
     public static void sleep(double secs) {
-        try {
-            java.lang.Thread.sleep((long)(secs * 1000));
-        }
-        catch (java.lang.InterruptedException e) {
-            throw new PyException(Py.KeyboardInterrupt, "interrupted sleep");
-        }
+	if (secs == 0) {
+	    // Conform to undocumented, or at least very underdocumented, but quite
+	    // reasonable behavior in CPython. See Alex Martelli's answer,
+	    // http://stackoverflow.com/a/790246/423006
+	    java.lang.Thread.yield();
+	} else {
+	    try {
+		java.lang.Thread.sleep((long)(secs * 1000));
+	    }
+	    catch (java.lang.InterruptedException e) {
+		throw new PyException(Py.KeyboardInterrupt, "interrupted sleep");
+	    }
+	}
     }
 
     // set by classDictInit()

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar  7 18:05:52 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 07 Mar 2015 17:05:52 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Improve_robustness_of_socke?=
 =?utf-8?q?t_and_SSL_support=2C_especially_error_handling?=
Message-ID: <20150307170552.1204.6390@psf.io>

https://hg.python.org/jython/rev/63711e6d4d41
changeset:   7604:63711e6d4d41
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sat Mar 07 10:05:18 2015 -0700
summary:
  Improve robustness of socket and SSL support, especially error handling

Note that SSL client connections need to defer the addition of the
inbound handler on the Netty pipeline until after the SSL connection
negotiates the handshake, much as has been done with server SSL. This
will require similar latching to avoid Netty bootstrap issues. In the
meantime, a short sleep (1 ms) seems to be an effective workaround
after doing a handshake. Addresses the issue raised in this email
thread: http://sourceforge.net/p/jython/mailman/message/33541508/ - in
this case, it does not appear to be a question of error handling (Java
exceptions appropriate wrapped as Python socket or SSL errors) as I
suggested in that email.

files:
  Lib/_socket.py              |  17 ++++++++++++-----
  Lib/_sslcerts.py            |   7 ++++---
  Lib/ssl.py                  |  13 ++++++++++++-
  Lib/test/regrtest.py        |   3 +--
  Lib/test/test_select_new.py |   6 +++---
  Lib/test/test_socket.py     |   1 +
  Lib/test/test_ssl.py        |  25 ++++++++++++++-----------
  7 files changed, 47 insertions(+), 25 deletions(-)


diff --git a/Lib/_socket.py b/Lib/_socket.py
--- a/Lib/_socket.py
+++ b/Lib/_socket.py
@@ -36,7 +36,7 @@
     # jarjar-ed version
     from org.python.netty.bootstrap import Bootstrap, ChannelFactory, ServerBootstrap
     from org.python.netty.buffer import PooledByteBufAllocator, Unpooled
-    from org.python.netty.channel import ChannelInboundHandlerAdapter, ChannelInitializer, ChannelOption
+    from org.python.netty.channel import ChannelException as NettyChannelException, ChannelInboundHandlerAdapter, ChannelInitializer, ChannelOption
     from org.python.netty.channel.nio import NioEventLoopGroup
     from org.python.netty.channel.socket import DatagramPacket
     from org.python.netty.channel.socket.nio import NioDatagramChannel, NioSocketChannel, NioServerSocketChannel
@@ -44,7 +44,7 @@
     # dev version from extlibs
     from io.netty.bootstrap import Bootstrap, ChannelFactory, ServerBootstrap
     from io.netty.buffer import PooledByteBufAllocator, Unpooled
-    from io.netty.channel import ChannelInboundHandlerAdapter, ChannelInitializer, ChannelOption
+    from io.netty.channel import ChannelException as NettyChannelException, ChannelInboundHandlerAdapter, ChannelInitializer, ChannelOption
     from io.netty.channel.nio import NioEventLoopGroup
     from io.netty.channel.socket import DatagramPacket
     from io.netty.channel.socket.nio import NioDatagramChannel, NioSocketChannel, NioServerSocketChannel
@@ -325,6 +325,8 @@
 
 
 def _map_exception(java_exception):
+    if isinstance(java_exception, NettyChannelException):
+        java_exception = java_exception.cause  # unwrap
     if isinstance(java_exception, SSLException) or isinstance(java_exception, CertificateException):
         cause = java_exception.cause
         if cause:
@@ -900,15 +902,18 @@
 
     # SERVER METHODS
     # Calling listen means this is a server socket
-
+    @raises_java_exception
     def listen(self, backlog):
         self.socket_type = SERVER_SOCKET
         self.child_queue = ArrayBlockingQueue(backlog)
         self.accepted_children = 1  # include the parent as well to simplify close logic
 
         b = ServerBootstrap()
-        self.parent_group = NioEventLoopGroup(_NUM_THREADS, DaemonThreadFactory("Jython-Netty-Parent-%s"))
-        self.child_group = NioEventLoopGroup(_NUM_THREADS, DaemonThreadFactory("Jython-Netty-Child-%s"))
+        try:
+            self.parent_group = NioEventLoopGroup(_NUM_THREADS, DaemonThreadFactory("Jython-Netty-Parent-%s"))
+            self.child_group = NioEventLoopGroup(_NUM_THREADS, DaemonThreadFactory("Jython-Netty-Child-%s"))
+        except IllegalStateException:
+            raise error(errno.EMFILE, "Cannot allocate thread pool for server socket")
         b.group(self.parent_group, self.child_group)
         b.channel(NioServerSocketChannel)
         b.option(ChannelOption.SO_BACKLOG, backlog)
@@ -1191,6 +1196,7 @@
     def fileno(self):
         return self
 
+    @raises_java_exception
     def setsockopt(self, level, optname, value):
         try:
             option, cast = _socket_options[self.proto][(level, optname)]
@@ -1203,6 +1209,7 @@
         if self.channel:
             _set_option(self.channel.config().setOption, option, cast_value)
 
+    @raises_java_exception
     def getsockopt(self, level, optname, buflen=None):
         # Pseudo options for interrogating the status of this socket
         if level == SOL_SOCKET:
diff --git a/Lib/_sslcerts.py b/Lib/_sslcerts.py
--- a/Lib/_sslcerts.py
+++ b/Lib/_sslcerts.py
@@ -28,9 +28,8 @@
     from org.bouncycastle.openssl import PEMKeyPair, PEMParser
     from org.bouncycastle.openssl.jcajce import JcaPEMKeyConverter
 
-log = logging.getLogger("ssl")
 
-
+log = logging.getLogger("_socket")
 Security.addProvider(BouncyCastleProvider())
 
 
@@ -91,7 +90,9 @@
                 elif isinstance(obj, X509CertificateHolder):
                     certs.append(cert_converter.getCertificate(obj))
 
-    assert private_key, "No private key loaded"
+    if not private_key:
+        from _socket import SSLError, SSL_ERROR_SSL
+        raise SSLError(SSL_ERROR_SSL, "No private key loaded")
     key_store = KeyStore.getInstance(KeyStore.getDefaultType())
     key_store.load(None, None)
     key_store.setKeyEntry(str(uuid.uuid4()), private_key, [], certs)
diff --git a/Lib/ssl.py b/Lib/ssl.py
--- a/Lib/ssl.py
+++ b/Lib/ssl.py
@@ -121,6 +121,13 @@
         if self.do_handshake_on_connect:
             self.do_handshake()
 
+    def connect_ex(self, addr):
+        log.debug("Connect SSL with handshaking %s", self.do_handshake_on_connect, extra={"sock": self._sock})
+        self._sock._connect(addr)
+        if self.do_handshake_on_connect:
+            self.do_handshake()
+        return self._sock.connect_ex(addr)
+
     def unwrap(self):
         self._sock.channel.pipeline().remove("ssl")
         self.ssl_handler.close()
@@ -151,7 +158,11 @@
                 self._sock.connect_handlers.append(SSLInitializer(self.ssl_handler))
 
         handshake = self.ssl_handler.handshakeFuture()
-        self._sock._handle_channel_future(handshake, "SSL handshake")
+        time.sleep(0.001)  # Necessary apparently for the handler to get into a good state
+        try:
+            self._sock._handle_channel_future(handshake, "SSL handshake")
+        except socket_error, e:
+            raise SSLError(SSL_ERROR_SSL, e.strerror)
 
     # Various pass through methods to the wrapped socket
 
diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py
--- a/Lib/test/regrtest.py
+++ b/Lib/test/regrtest.py
@@ -1269,6 +1269,7 @@
         # Nonreliable tests
         test_asynchat
         test_asyncore
+        test_select_new
 
         # Command line testing is hard for Jython to do, but revisit
         test_cmd_line_script
@@ -1279,9 +1280,7 @@
         test_poplib
         test_smtplib
         test_socket_ssl
-        test_socketserver
         test_telnetlib
-        test_timeout
 
         test_sys_setprofile  # revisit for GC
         test_sys_settrace    # revisit for line jumping
diff --git a/Lib/test/test_select_new.py b/Lib/test/test_select_new.py
--- a/Lib/test/test_select_new.py
+++ b/Lib/test/test_select_new.py
@@ -36,8 +36,8 @@
         self.server_addr = self.server_socket.getsockname()
         try:
             self.server_socket.accept()
-        except socket.error:
-            pass
+        except socket.error, e:
+            pass  # at this point, always gets EWOULDBLOCK - nothing to accept
 
     def select_acceptable(self):
         return select.select([self.server_socket], [self.server_socket], [], SELECT_TIMEOUT)[0]
@@ -151,7 +151,7 @@
         if result == errno.EISCONN:
             self.connected = True
         else:
-            assert result == errno.EINPROGRESS, \
+            assert result in [errno.EINPROGRESS, errno.ENOTCONN], \
                 "connect_ex returned %s (%s)" % (result, errno.errorcode.get(result, "Unknown errno"))
 
     def finish_connect(self):
diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py
--- a/Lib/test/test_socket.py
+++ b/Lib/test/test_socket.py
@@ -172,6 +172,7 @@
             self._assert_no_pending_threads(self.srv.group, "Server thread pool")
 
         if msg:
+            print "Got this message=%s %r" % (type(msg), msg)
             self.fail("msg={}".format(msg))
 
             
diff --git a/Lib/test/test_ssl.py b/Lib/test/test_ssl.py
--- a/Lib/test/test_ssl.py
+++ b/Lib/test/test_ssl.py
@@ -164,7 +164,8 @@
                         (s, t))
 
     def test_ciphers(self):
-        if not test_support.is_resource_enabled('network'):
+        if not test_support.is_resource_enabled('network') or test_support.is_jython:
+            # see note on Jython support in test_main()
             return
         remote = ("svn.python.org", 443)
         with test_support.transient_internet(remote[0]):
@@ -209,13 +210,13 @@
 
     def test_connect(self):
         with test_support.transient_internet("svn.python.org"):
-            s = ssl.wrap_socket(socket.socket(socket.AF_INET),
-                                cert_reqs=ssl.CERT_NONE)
-            s.connect(("svn.python.org", 443))
-            c = s.getpeercert()
-            if c:
-                self.fail("Peer cert %s shouldn't be here!")
-            s.close()
+            # s = ssl.wrap_socket(socket.socket(socket.AF_INET),
+            #                     cert_reqs=ssl.CERT_NONE)
+            # s.connect(("svn.python.org", 443))
+            # c = s.getpeercert()
+            # if c:
+            #     self.fail("Peer cert %s shouldn't be here!")
+            # s.close()
 
             # this should fail because we have no verification certs
             s = ssl.wrap_socket(socket.socket(socket.AF_INET),
@@ -1377,10 +1378,12 @@
 
     tests = [BasicTests, BasicSocketTests]
 
-    if test_support.is_resource_enabled('network'):
+    if test_support.is_resource_enabled('network') and not test_support.is_jython:
+        # These tests need to be updated since they rely on CERT_NONE
+        # and in certain cases unavailable network resources
         tests.append(NetworkedTests)
 
-    if _have_threads:
+    if _have_threads and not test_support.is_jython:
         thread_info = test_support.threading_setup()
         if thread_info and test_support.is_resource_enabled('network'):
             tests.append(ThreadedTests)
@@ -1388,7 +1391,7 @@
     try:
         test_support.run_unittest(*tests)
     finally:
-        if _have_threads:
+        if _have_threads and not test_support.is_jython:
             test_support.threading_cleanup(*thread_info)
 
 if __name__ == "__main__":

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar  7 20:13:01 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 07 Mar 2015 19:13:01 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Upgrade_JLine_to_2=2E12=2E1?=
 =?utf-8?q?_to_support_keyboard_layouts_using_AltGr?=
Message-ID: <20150307191301.16661.35673@psf.io>

https://hg.python.org/jython/rev/e050becb7e28
changeset:   7605:e050becb7e28
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sat Mar 07 11:59:46 2015 -0700
summary:
  Upgrade JLine to 2.12.1 to support keyboard layouts using AltGr

Fixes http://bugs.jython.org/issue2275

files:
  build.xml                |    4 ++--
  extlibs/jline-2.12.1.jar |  Bin 
  extlibs/jline-2.12.jar   |  Bin 
  3 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/build.xml b/build.xml
--- a/build.xml
+++ b/build.xml
@@ -177,7 +177,7 @@
             <pathelement path="${extlibs.dir}/jnr-netdb-1.1.4.jar"/>
             <pathelement path="${extlibs.dir}/jnr-posix-3.0.9.jar"/>
             <pathelement path="${extlibs.dir}/jnr-constants-0.8.6.jar"/>
-            <pathelement path="${extlibs.dir}/jline-2.12.jar"/>
+            <pathelement path="${extlibs.dir}/jline-2.12.1.jar"/>
             <pathelement path="${extlibs.dir}/netty-buffer-4.0.25.Final.jar"/>
             <pathelement path="${extlibs.dir}/netty-codec-4.0.25.Final.jar"/>
             <pathelement path="${extlibs.dir}/netty-common-4.0.25.Final.jar"/>
@@ -628,7 +628,7 @@
             <rule pattern="org.apache.xerces.**" result="org.python.apache.xerces. at 1"/>
             <rule pattern="org.apache.wml.**" result="org.python.apache.wml. at 1"/>
             <rule pattern="org.apache.html.**" result="org.python.apache.html. at 1"/>
-            <zipfileset src="extlibs/jline-2.12.jar"/>
+            <zipfileset src="extlibs/jline-2.12.1.jar"/>
             <rule pattern="jline.**" result="org.python.jline. at 1"/>
             <manifest>
                 <attribute name="Main-Class" value="org.python.util.jython" />
diff --git a/extlibs/jline-2.12.1.jar b/extlibs/jline-2.12.1.jar
new file mode 100644
index e69de29bb2d1d6434b8b29ae775ad8c2e48c5391..00dffb02ea4c3c0b15df8b3ab4727124dbb02dc8
GIT binary patch
[stripped]
diff --git a/extlibs/jline-2.12.jar b/extlibs/jline-2.12.jar
deleted file mode 100644
index 41676a1e210dec6f80cb730830b69dab202a0852..e69de29bb2d1d6434b8b29ae775ad8c2e48c5391
GIT binary patch
[stripped]

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Tue Mar 10 02:21:17 2015
From: jython-checkins at python.org (jim.baker)
Date: Tue, 10 Mar 2015 01:21:17 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Synchronize_PySystemStateCl?=
 =?utf-8?q?oser=2C_FinalizeTrigger=23runFinalizer?=
Message-ID: <20150310012117.8083.92763@psf.io>

https://hg.python.org/jython/rev/f79dc13778f9
changeset:   7606:f79dc13778f9
user:        Jim Baker <jim.baker at rackspace.com>
date:        Mon Mar 09 19:21:13 2015 -0600
summary:
  Synchronize PySystemStateCloser, FinalizeTrigger#runFinalizer

Avoid deadlock at PySystemState shutdown by having
PySystemStateCloser's methods and FinalizeTrigger#runFinalizer both
synchronize on the monitor for PySystemStateCloser.class

Fixes http://bugs.jython.org/issue2280

files:
  src/org/python/core/PySystemState.java                |  103 +++++----
  src/org/python/core/finalization/FinalizeTrigger.java |   32 +-
  2 files changed, 75 insertions(+), 60 deletions(-)


diff --git a/src/org/python/core/PySystemState.java b/src/org/python/core/PySystemState.java
--- a/src/org/python/core/PySystemState.java
+++ b/src/org/python/core/PySystemState.java
@@ -1546,7 +1546,7 @@
 
     public void close() { cleanup(); }
 
-    private static class PySystemStateCloser {
+    public static class PySystemStateCloser {
 
         private final Set<Callable<Void>> resourceClosers = new LinkedHashSet<Callable<Void>>();
         private volatile boolean isCleanup = false;
@@ -1568,57 +1568,64 @@
             }
         }
 
-        private synchronized void registerCloser(Callable<Void> closer) {
-            if (!isCleanup) {
-                resourceClosers.add(closer);
+        private void registerCloser(Callable<Void> closer) {
+            synchronized (PySystemStateCloser.class) {
+                if (!isCleanup) {
+                    resourceClosers.add(closer);
+                }
             }
         }
 
-        private synchronized boolean unregisterCloser(Callable<Void> closer) {
-            return resourceClosers.remove(closer);
+        private boolean unregisterCloser(Callable<Void> closer) {
+            synchronized (PySystemStateCloser.class) {
+                return resourceClosers.remove(closer);
+            }
         }
 
         private synchronized void cleanup() {
-            if (isCleanup) {
-                return;
+            synchronized (PySystemStateCloser.class) {
+                if (isCleanup) {
+                    return;
+                }
+                isCleanup = true;
+
+                // close this thread so we can unload any associated classloaders in cycle
+                // with this instance
+                if (shutdownHook != null) {
+                    try {
+                        Runtime.getRuntime().removeShutdownHook(shutdownHook);
+                    } catch (IllegalStateException e) {
+                        // JVM is already shutting down, so we cannot remove this shutdown hook anyway
+                    }
+                }
+
+                // Close the listed resources (and clear the list)
+                runClosers();
+                resourceClosers.clear();
+
+                // XXX Not sure this is ok, but it makes repeat testing possible.
+                // Re-enable the management of resource closers
+                isCleanup = false;
             }
-            isCleanup = true;
+        }
+        private void runClosers() {
+            synchronized (PySystemStateCloser.class) {
+                // resourceClosers can be null in some strange cases
+                if (resourceClosers != null) {
+                    /*
+                     * Although a Set, the container iterates in the order closers were added. Make a Deque
+                     * of it and deal from the top.
+                     */
+                    LinkedList<Callable<Void>> rc = new LinkedList<Callable<Void>>(resourceClosers);
+                    Iterator<Callable<Void>> iter = rc.descendingIterator();
 
-            // close this thread so we can unload any associated classloaders in cycle
-            // with this instance
-            if (shutdownHook != null) {
-                try {
-                    Runtime.getRuntime().removeShutdownHook(shutdownHook);
-                } catch (IllegalStateException e) {
-                    // JVM is already shutting down, so we cannot remove this shutdown hook anyway
-                }
-            }
-
-            // Close the listed resources (and clear the list)
-            runClosers();
-            resourceClosers.clear();
-
-            // XXX Not sure this is ok, but it makes repeat testing possible.
-            // Re-enable the management of resource closers
-            isCleanup = false;
-        }
-
-        private synchronized void runClosers() {
-            // resourceClosers can be null in some strange cases
-            if (resourceClosers != null) {
-            /*
-             * Although a Set, the container iterates in the order closers were added. Make a Deque
-             * of it and deal from the top.
-             */
-                LinkedList<Callable<Void>> rc = new LinkedList<Callable<Void>>(resourceClosers);
-                Iterator<Callable<Void>> iter = rc.descendingIterator();
-
-                while (iter.hasNext()) {
-                    Callable<Void> callable = iter.next();
-                    try {
-                        callable.call();
-                    } catch (Exception e) {
-                        // just continue, nothing we can do
+                    while (iter.hasNext()) {
+                        Callable<Void> callable = iter.next();
+                        try {
+                            callable.call();
+                        } catch (Exception e) {
+                            // just continue, nothing we can do
+                        }
                     }
                 }
             }
@@ -1639,9 +1646,11 @@
         private class ShutdownCloser implements Runnable {
 
             @Override
-            public synchronized void run() {
-                runClosers();
-                resourceClosers.clear();
+            public void run() {
+                synchronized (PySystemStateCloser.class) {
+                    runClosers();
+                    resourceClosers.clear();
+                }
             }
         }
 
diff --git a/src/org/python/core/finalization/FinalizeTrigger.java b/src/org/python/core/finalization/FinalizeTrigger.java
--- a/src/org/python/core/finalization/FinalizeTrigger.java
+++ b/src/org/python/core/finalization/FinalizeTrigger.java
@@ -3,6 +3,7 @@
 import org.python.core.PyObject;
 import org.python.core.JyAttribute;
 import org.python.core.Py;
+import org.python.core.PySystemState;
 import org.python.modules.gc;
 
 /**
@@ -76,22 +77,27 @@
     }
 
     public static void runFinalizer(PyObject toFinalize, boolean runBuiltinOnly) {
-        if (!runBuiltinOnly) {
-            if (toFinalize instanceof FinalizablePyObjectDerived) {
+        synchronized (PySystemState.PySystemStateCloser.class) {
+            if (!runBuiltinOnly) {
+                if (toFinalize instanceof FinalizablePyObjectDerived) {
+                    try {
+                        ((FinalizablePyObjectDerived) toFinalize).__del_derived__();
+                    } catch (Exception e) {
+                    }
+                } else if (toFinalize instanceof FinalizablePyObject) {
+                    try {
+                        ((FinalizablePyObject) toFinalize).__del__();
+                    } catch (Exception e) {
+                    }
+                }
+            }
+            if (toFinalize instanceof FinalizableBuiltin) {
                 try {
-                    ((FinalizablePyObjectDerived) toFinalize).__del_derived__();
-                } catch (Exception e) {}
-            } else if (toFinalize instanceof FinalizablePyObject) {
-                try {
-                    ((FinalizablePyObject) toFinalize).__del__();
-                } catch (Exception e) {}
+                    ((FinalizableBuiltin) toFinalize).__del_builtin__();
+                } catch (Exception e) {
+                }
             }
         }
-        if (toFinalize instanceof FinalizableBuiltin) {
-            try {
-                ((FinalizableBuiltin) toFinalize).__del_builtin__();
-            } catch (Exception e) {}
-        }
     }
 
     public static void appendFinalizeTriggerForBuiltin(PyObject obj) {

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Tue Mar 10 18:10:39 2015
From: jython-checkins at python.org (jim.baker)
Date: Tue, 10 Mar 2015 17:10:39 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Generalize_adding_special_s?=
 =?utf-8?b?bG90IGRlc2NyaXB0b3JzIChfX2RpY3RfXywgX193ZWFrcmVmX18p?=
Message-ID: <20150310171037.9682.92358@psf.io>

https://hg.python.org/jython/rev/4d28c47997de
changeset:   7607:4d28c47997de
user:        Jim Baker <jim.baker at rackspace.com>
date:        Tue Mar 10 11:10:34 2015 -0600
summary:
  Generalize adding special slot descriptors (__dict__, __weakref__)

Jython used to be too strict, and would throw TypeError if the special
slot descriptors of __dict__ and __weakref__ were used in a
subclass. With this change, CPython is stricter than Jython (see tests
in test_slots_jy for specifics). However Jython's underlying object
model is based on Java, and that give us more flexibility. In
particular, it's now possible to add a __dict__ slot descriptor to a
subclass of Java classes, such as java.util.HashMap.

Fixes http://bugs.jython.org/issue2272 and adds support for Werkzeug

files:
  Lib/test/test_slots_jy.py       |  98 ++++++++++++++++++++-
  src/org/python/core/PyType.java |  34 ++++--
  2 files changed, 120 insertions(+), 12 deletions(-)


diff --git a/Lib/test/test_slots_jy.py b/Lib/test/test_slots_jy.py
--- a/Lib/test/test_slots_jy.py
+++ b/Lib/test/test_slots_jy.py
@@ -3,6 +3,7 @@
 Made for Jython.
 """
 from test import test_support
+from java.util import HashMap
 import unittest
 
 # The strict tests fail on PyPy (but work on CPython and Jython).
@@ -134,10 +135,105 @@
         self.assert_(hasattr(Foo, '__weakref__'))
 
 
+class SpecialSlotsBaseTestCase(unittest.TestCase):
+    
+    # Tests for http://bugs.jython.org/issue2272, including support for
+    # werkzeug.local.LocalProxy
+    
+    def make_class(self, base, slot):
+        class C(base):
+            __slots__ = (slot)
+            if slot == "__dict__":
+                @property
+                def __dict__(self):
+                    return {"x": 42, "y": 47}
+                def __getattr__(self, name):
+                    try:
+                        return self.__dict__[name]
+                    except KeyError:
+                        raise AttributeError("%r object has no attribute %r" % (
+                            self.__class__.__name__, name))
+        return C
+
+    def test_dict_slot(self):
+        C = self.make_class(object, "__dict__")
+        c = C()
+        self.assertIn("__dict__", dir(c))
+        self.assertIn("x", dir(c))
+        self.assertIn("y", dir(c))
+        self.assertEqual(c.__dict__.get("x"), 42)
+        self.assertEqual(c.x, 42)
+        self.assertEqual(c.y, 47)
+        with self.assertRaisesRegexp(AttributeError, r"'C' object has no attribute 'z'"):
+            c.z
+
+    def test_dict_slot_str(self):
+        # Unlike CPython, Jython does not arbitrarily limit adding
+        # __dict__ slot to str and other types that are not object
+        C = self.make_class(str, "__dict__")
+        c = C("abc123")
+        self.assertTrue(c.startswith("abc"))
+        self.assertIn("__dict__", dir(c))
+        self.assertIn("x", dir(c))
+        self.assertIn("y", dir(c))
+        self.assertEqual(c.__dict__.get("x"), 42)
+        self.assertEqual(c.x, 42)
+        self.assertEqual(c.y, 47)
+        with self.assertRaisesRegexp(AttributeError, r"'C' object has no attribute 'z'"):
+            c.z
+
+    def test_dict_slot_subclass(self):
+        # Unlike CPython, Jython does not arbitrarily limit adding __dict__ slot to subtypes
+        class B(object):
+            @property
+            def w(self):
+                return 23
+        C = self.make_class(B, "__dict__")
+        c = C()
+        self.assertIn("__dict__", dir(c))
+        self.assertIn("x", dir(c))
+        self.assertIn("y", dir(c))
+        self.assertEqual(c.__dict__.get("x"), 42)
+        self.assertEqual(c.x, 42)
+        self.assertEqual(c.y, 47)
+        with self.assertRaisesRegexp(AttributeError, r"'C' object has no attribute 'z'"):
+            c.z
+        self.assertEqual(c.w, 23)
+
+    def test_dict_slot_subclass_java_hashmap(self):
+        C = self.make_class(HashMap, "__dict__")
+        # has everything in a HashMap, including Python semantic equivalence
+        c = C({"a": 1, "b": 2})
+        self.assertTrue(c.containsKey("a"))
+        self.assertEqual(sorted(c.iteritems()), [("a", 1), ("b", 2)])
+        # but also has a __dict__ slot for further interesting ;) possibilities
+        self.assertIn("__dict__", dir(c))
+        self.assertIn("x", dir(c))
+        self.assertIn("y", dir(c))
+        self.assertEqual(c.__dict__.get("x"), 42)
+        self.assertEqual(c.x, 42)
+        self.assertEqual(c.y, 47)
+        with self.assertRaisesRegexp(AttributeError, r"'C' object has no attribute 'z'"):
+            c.z
+
+    def test_weakref_slot(self):
+        self.assertNotIn("__weakref__", dir(object()))
+        self.assertIn("__weakref__", dir(self.make_class(object, "__weakref__")()))
+        class B(object):
+            pass
+        self.assertIn("__weakref__", dir(B()))
+        self.assertIn("__weakref__", dir(self.make_class(B, "__weakref__")()))
+        self.assertNotIn("__weakref__", dir("abc"))
+        self.assertIn("__weakref__", dir(self.make_class(str, "__weakref__")()))
+        self.assertNotIn("__weakref__", dir(HashMap()))
+        self.assertIn("__weakref__", dir(self.make_class(HashMap, "__weakref__")()))
+
+
 def test_main():
     test_support.run_unittest(SlottedTestCase,
                               SlottedWithDictTestCase,
-                              SlottedWithWeakrefTestCase)
+                              SlottedWithWeakrefTestCase,
+                              SpecialSlotsBaseTestCase)
 
 
 if __name__ == '__main__':
diff --git a/src/org/python/core/PyType.java b/src/org/python/core/PyType.java
--- a/src/org/python/core/PyType.java
+++ b/src/org/python/core/PyType.java
@@ -102,7 +102,7 @@
     /** Mapping of Java classes to their PyTypes. */
     private static Map<Class<?>, PyType> class_to_type;
     private static Set<PyType> exposedTypes;
-    
+
     /** Mapping of Java classes to their TypeBuilders. */
     private static Map<Class<?>, TypeBuilder> classToBuilder;
 
@@ -255,19 +255,31 @@
 
                 if (slotName.equals("__dict__")) {
                     if (!mayAddDict || wantDict) {
-                        throw Py.TypeError("__dict__ slot disallowed: we already got one");
+                        // CPython is stricter here, but this seems arbitrary. To reproduce CPython
+                        // behavior
+                        // if (base != PyObject.TYPE) {
+                        //     throw Py.TypeError("__dict__ slot disallowed: we already got one");
+                        // }
+                    } else {
+                        wantDict = true;
+                        continue;
                     }
-                    wantDict = true;
                 } else if (slotName.equals("__weakref__")) {
-                    if (!mayAddWeak || wantWeak) {
-                        throw Py.TypeError("__weakref__ slot disallowed: we already got one");
+                    if ((!mayAddWeak || wantWeak) && base != PyObject.TYPE) {
+                        // CPython is stricter here, but this seems arbitrary. To reproduce CPython
+                        // behavior
+                        // if (base != PyObject.TYPE) {
+                        //     throw Py.TypeError("__weakref__ slot disallowed: we already got one");
+                        // }
+                    } else {
+                        wantWeak = true;
+                        continue;
                     }
-                    wantWeak = true;
-                } else {
-                    slotName = mangleName(name, slotName);
-                    if (dict.__finditem__(slotName) == null) {
-                        dict.__setitem__(slotName, new PySlot(this, slotName, numSlots++));
-                    }
+                }
+
+                slotName = mangleName(name, slotName);
+                if (dict.__finditem__(slotName) == null) {
+                    dict.__setitem__(slotName, new PySlot(this, slotName, numSlots++));
                 }
             }
 

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 13 05:15:03 2015
From: jython-checkins at python.org (stefan.richthofer)
Date: Fri, 13 Mar 2015 04:15:03 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Added_detailed_javadoc_to_g?=
 =?utf-8?q?c-module=3B_fixed_minor_gc-issues_and_added?=
Message-ID: <20150313041503.30204.73573@psf.io>

https://hg.python.org/jython/rev/9c59fdd3b79e
changeset:   7608:9c59fdd3b79e
user:        Stefan Richthofer <stefan.richthofer at gmx.de>
date:        Fri Mar 13 05:14:46 2015 +0100
summary:
  Added detailed javadoc to gc-module; fixed minor gc-issues and added
a test for third-party finalizers; implemented Traverseproc for
zipimporter

files:
  Lib/test/test_gc.py                                   |    8 +
  Lib/test/test_gc_jy.py                                |   67 +-
  src/org/python/core/finalization/FinalizeTrigger.java |   39 +-
  src/org/python/modules/gc.java                        |  512 ++++++++-
  src/org/python/modules/zipimport/zipimporter.java     |   22 +-
  tests/java/javatests/GCTestHelper.java                |    2 +-
  6 files changed, 533 insertions(+), 117 deletions(-)


diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py
--- a/Lib/test/test_gc.py
+++ b/Lib/test/test_gc.py
@@ -313,6 +313,14 @@
         gc.collect(2)
         assertEqual(gc.get_count(), (0, 0, 0))
 
+    @unittest.skipIf(test_support.is_jython,
+        '''
+        While this test passes in Jython, it leads to internal
+        allocation failures because of the massive referencing
+        in this test. To keep the JVM-process healthy and to
+        avoid subsequent failures due to bad conditions caused
+        by this test, we skip it for now.
+        ''')
     def test_trashcan(self):
         class Ouch:
             n = 0
diff --git a/Lib/test/test_gc_jy.py b/Lib/test/test_gc_jy.py
--- a/Lib/test/test_gc_jy.py
+++ b/Lib/test/test_gc_jy.py
@@ -113,6 +113,22 @@
         'CPython has no gc preprocess and postprocess features')
 class GCTests_Jy_preprocess_and_postprocess(unittest.TestCase):
 
+    @classmethod
+    def setUpClass(cls):
+        #Jython-specific block:
+        try:
+            cls.savedJythonGCFlags = gc.getJythonGCFlags()
+            gc.setMonitorGlobal(True)
+        except Exception:
+            pass
+
+    @classmethod
+    def tearDownClass(cls):
+        try:
+            gc.setJythonGCFlags(cls.savedJythonGCFlags)
+        except Exception:
+            pass
+
     def test_finalization_preprocess_and_postprocess(self):
         #Note that this test is done here again (already was in another class
         #in this module), to see that everything works as it should also with
@@ -167,9 +183,6 @@
         gc.unregisterPostFinalizationProcess(postPr)
 
     def test_with_extern_NonPyObjectFinalizer_that_notifies_gc(self):
-        #Note that this test is done here again (already was in another class
-        #in this module), to see that everything works as it should also with
-        #a different flag-context.
         comments = []
         class A:
             def __init__(self, index):
@@ -238,6 +251,8 @@
         except Exception:
             pass
 
+    #Tests from GCTests_Jy_preprocess_and_postprocess are repeated here
+    #without monitoring.
     def test_finalization_preprocess_and_postprocess(self):
         #Note that this test is done here again (already was in another class
         #in this module), to see that everything works as it should also with
@@ -290,8 +305,54 @@
         gc.unregisterPreFinalizationProcess(prePr)
         gc.unregisterPostFinalizationProcess(postPr)
 
+    def test_with_extern_NonPyObjectFinalizer_that_notifies_gc(self):
+        comments = []
+        class A:
+            def __init__(self, index):
+                self.index = index
+
+            def __del__(self):
+                comments.append("A_del_"+str(self.index))
+
+        class PreProcess(Runnable):
+            preCount = 0
+            def run(self):
+                PreProcess.preCount += 1
+
+        class PostProcess(Runnable):
+            postCount = 0
+            def run(self):
+                PostProcess.postCount += 1
+
+        prePr = PreProcess()
+        postPr = PostProcess()
+        time.sleep(1) #   <- to avoid that the newly registered processes
+                      #      become subject to previous run (remember: We
+                      #      are not in monitor-mode, i.e. gc runs async.
+        gc.registerPreFinalizationProcess(prePr)
+        gc.registerPostFinalizationProcess(postPr)
+        for i in range(4):
+            f = A(i)
+            del f
+        #NastyFinalizer would cause this test occasionally to fail
+        externFinalizer = GCTestHelper.NotSoNastyFinalizer()
+        del externFinalizer
+        for i in range(4, 8):
+            f = A(i)
+            del f
+        System.gc()
+        #we wait a bit longer here, since PostProcess runs asynchronous
+        #and must wait for the finalizer of A
+        time.sleep(4)
+        self.assertEqual(len(comments), 8)
+        self.assertEqual(PreProcess.preCount, 1)
+        self.assertEqual(PostProcess.postCount, 1)
+        comments = []
+        gc.unregisterPreFinalizationProcess(prePr)
+        gc.unregisterPostFinalizationProcess(postPr)
 
     def test_delayedFinalization(self):
+        #time.sleep(2)
         resurrect = []
         comments = []
 
diff --git a/src/org/python/core/finalization/FinalizeTrigger.java b/src/org/python/core/finalization/FinalizeTrigger.java
--- a/src/org/python/core/finalization/FinalizeTrigger.java
+++ b/src/org/python/core/finalization/FinalizeTrigger.java
@@ -2,7 +2,6 @@
 
 import org.python.core.PyObject;
 import org.python.core.JyAttribute;
-import org.python.core.Py;
 import org.python.core.PySystemState;
 import org.python.modules.gc;
 
@@ -131,27 +130,27 @@
     }
 
     protected boolean isCyclic() {
-    	gc.CycleMarkAttr cm = (gc.CycleMarkAttr)
-    			JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
+        gc.CycleMarkAttr cm = (gc.CycleMarkAttr)
+                JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
         if (cm != null && cm.isCyclic()) {
             return true;
         } else {
             gc.markCyclicObjects(toFinalize, (flags & NOT_FINALIZABLE_FLAG) == 0);
             cm = (gc.CycleMarkAttr)
-        			JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
+                    JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
             return cm != null && cm.isCyclic();
         }
     }
 
     protected boolean isUncollectable() {
-    	gc.CycleMarkAttr cm = (gc.CycleMarkAttr)
-    			JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
+        gc.CycleMarkAttr cm = (gc.CycleMarkAttr)
+                JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
         if (cm != null && cm.isUncollectable()) {
             return true;
         } else {
             gc.markCyclicObjects(toFinalize, (flags & NOT_FINALIZABLE_FLAG) == 0);
             cm = (gc.CycleMarkAttr)
-        			JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
+                    JyAttribute.getAttr(toFinalize, JyAttribute.GC_CYCLE_MARK_ATTR);
             return cm != null && cm.isUncollectable();
         }
     }
@@ -167,13 +166,13 @@
                     runFinalizer(toFinalize, (flags & ONLY_BUILTIN_FLAG) != 0);
                 }
             } else {
-            	if ((flags & NOT_FINALIZABLE_FLAG) == 0) {
-            		runFinalizer(toFinalize, (flags & ONLY_BUILTIN_FLAG) != 0);
-            	}
+                if ((flags & NOT_FINALIZABLE_FLAG) == 0) {
+                    runFinalizer(toFinalize, (flags & ONLY_BUILTIN_FLAG) != 0);
+                }
             }
             if ((gc.getJythonGCFlags() & gc.VERBOSE_FINALIZE) != 0) {
-        		gc.writeDebug("gc", "finalization of "+toFinalize);
-        	}
+                gc.writeDebug("gc", "finalization of "+toFinalize);
+            }
             if (saveGarbage == 1 || (saveGarbage == 0 &&
                     (gc.get_debug() & gc.DEBUG_SAVEALL) != 0 && isCyclic())) {
                 if ((flags & NOT_FINALIZABLE_FLAG) == 0) {
@@ -185,14 +184,14 @@
                 }
                 gc.garbage.add(toFinalize);
                 if ((gc.getJythonGCFlags() & gc.VERBOSE_FINALIZE) != 0) {
-                	gc.writeDebug("gc", toFinalize+" added to garbage.");
-            	}
+                    gc.writeDebug("gc", toFinalize+" added to garbage.");
+                }
             }
         }
         if ((flags & NOTIFY_GC_FLAG) != 0) {
-        	if ((gc.getJythonGCFlags() & gc.VERBOSE_FINALIZE) != 0) {
-        		gc.writeDebug("gc", "notify finalization of "+toFinalize);
-        	}
+            if ((gc.getJythonGCFlags() & gc.VERBOSE_FINALIZE) != 0) {
+                gc.writeDebug("gc", "notify finalization of "+toFinalize);
+            }
             gc.notifyFinalize(toFinalize);
             flags &= ~NOTIFY_GC_FLAG;
         }
@@ -202,9 +201,9 @@
         flags |= FINALIZED_FLAG;
         gc.notifyPreFinalization();
         if (gc.delayedFinalizationEnabled() && toFinalize != null) {
-        	if ((gc.getJythonGCFlags() & gc.VERBOSE_FINALIZE) != 0) {
-        		gc.writeDebug("gc", "delayed finalization for "+toFinalize);
-        	}
+            if ((gc.getJythonGCFlags() & gc.VERBOSE_FINALIZE) != 0) {
+                gc.writeDebug("gc", "delayed finalization for "+toFinalize);
+            }
             gc.registerForDelayedFinalization(toFinalize);
         } else {
             performFinalization();
diff --git a/src/org/python/modules/gc.java b/src/org/python/modules/gc.java
--- a/src/org/python/modules/gc.java
+++ b/src/org/python/modules/gc.java
@@ -26,8 +26,149 @@
 import org.python.core.finalization.FinalizeTrigger;
 import org.python.modules._weakref.GlobalRef;
 
-import com.sun.management.GarbageCollectionNotificationInfo;
+//These imports belong to the out-commented section on MXBean-based
+//GC-sync far below. That section is kept to document this failed
+//approach and allow easy reproduction of this failure.
+//import java.lang.management.*;
+//import javax.management.*;
+//import javax.management.openmbean.*;
 
+/**
+ * <p>
+ * In Jython, the gc-module notably differs from that in CPython.
+ * This comes from the different ways Jython and CPython perform
+ * garbage-collection. While CPython's garbage collection is based on
+ * <a href="http://en.wikipedia.org/wiki/Reference_counting" target="_blank">
+ * reference-counting</a>, Jython is backed by Java's gc, which is
+ * based on a
+ * <a href="http://en.wikipedia.org/wiki/Tracing_garbage_collection" target="_blank">
+ * mark-and-sweep-approach</a>.
+ * </p>
+ * <p>
+ * This difference becomes most notable if finalizers are involved that perform resurrection.
+ * While the resurrected object itself behaves rather similar between Jython and CPython,
+ * things are more delicate with objects that are reachable (i.e. strongly referenced)
+ * via the resurrected object exclusively.
+ * While in CPython such objects do not get their finalizers called, Jython/Java would
+ * call all their finalizers. That is because Java detects the whole unreachable subgraph
+ * as garbage and thus calls all their finalizers without any chance of direct intervention.
+ * CPython instead detects the unreachable object and calls its finalizer, which makes the
+ * object reachable again. Then all other objects are reachable from it and CPython does not
+ * treat them as garbage and does not call their finalizers at all.
+ * This further means that in Jython weak references to such indirectly resurrected objects
+ * break, while these persist in CPython.
+ * </p>
+ * <p>
+ * As of Jython 2.7, the gc-module offers some options to emulate CPython-behavior.
+ * Especially see the flags {@link #PRESERVE_WEAKREFS_ON_RESURRECTION},
+ * {@link #DONT_FINALIZE_RESURRECTED_OBJECTS} and {@link #DONT_FINALIZE_CYCLIC_GARBAGE}
+ * for this.
+ * </p>
+ * <p>
+ * Another difference is that CPython's gc-module offers some debug-features like counting
+ * of collected cyclic trash, which are hard to support by Jython. As of Jython 2.7 the
+ * introduction of a traverseproc-mechanism (c.f. {@link org.python.core.Traverseproc})
+ * made support of these features feasible. As support of these features comes
+ * with a significant emulation-cost, one must explicitly tell gc to perform this.
+ * To make objects subject to cyclic trash-counting, these objects must be gc-monitored in
+ * Jython. See {@link #monitorObject(PyObject)}, {@link #unmonitorObject(PyObject)},
+ * {@link #MONITOR_GLOBAL} and {@link #stopMonitoring()} for this.<br>
+ * If at least one object is gc-monitored, {@link #collect()} works synchronously in the
+ * sense that it blocks until all gc-monitored objects that are garbage actually have been
+ * collected and had their finalizers called and completed. {@link #collect()} will report
+ * the number of collected objects in the same manner as in CPython, i.e. counts only those
+ * that participate in reference cycles. This allows a unified test-implementation across
+ * Jython and CPython (which applies to most tests in test_gc.py). If not any object is
+ * gc-monitored, {@link #collect()} just delegates to {@link java.lang.System.gc()}, runs
+ * asynchronously (i.e. non-blocking) and returns {@link #UNKNOWN_COUNT}.
+ * See also {@link #DEBUG_SAVEALL} for a useful gc-debugging feature that is supported by
+ * Jython from version 2.7 onwards.
+ * </p>
+ * <p>
+ * Implementing all these features in Jython involved a lot of synchronization logic.
+ * While care was taken to implement this without using timeouts as far as possible and
+ * rely on locks, states and system/hardware independent synchronization techniques,
+ * this was not entirely feasible.<br>
+ * The aspects that were only feasible using a timeout are waiting for gc to enqueue all
+ * collected objects (i.e. weak references to monitored objects that were gc'ed) to the
+ * reference queue and waiting for gc to run all PyObject-finalizers.
+ * </p>
+ * <p>
+ * Waiting for trash could in theory be strictly synchronized by using {@code MXBean}s, i.e.
+ * <a href="https://docs.oracle.com/javase/7/docs/jre/api/management/extension/index.html?com/sun/management/GcInfo.html"
+ * target="_blank">GarbageCollectionNotificationInfo</a> and related API.
+ * However, experiments showed that the arising GC-notifications do not reliably indicate
+ * when enqueuing was done for a specific GC-run. We kept the experimental implementation
+ * in source-code comments to allow easy reproducibility of this issue. (Note that out-commented
+ * code contradicts Jython-styleguide, but this one - however - is needed to document this
+ * infeasible approach and is explicitly declared accordingly).<br>
+ * But how <b>is</b> sync done now?
+ * We insert a sentinel before running gc and wait until this sentinel was collected.
+ * Timestamps are taken to give us an idea at which time-scales the gc of the current JVM
+ * performs. We then wait until twice the measured time (i.e. duration from call to
+ * {@link java.lang.System#gc()} until the sentinel reference was enqueued) has passed after
+ * the last reference was enqueued by gc. While this approach is not entirely safe in theory,
+ * it passes all tests on various systems and machines we had available for testing so far.
+ * We consider it more robust than a fixed-length timeout and regard it the best known feasible
+ * compromise to emulate synchronous gc-runs in Java.
+ * </p>
+ * <p>
+ * The other timing-based synchronization issue - waiting for finalizers to run - is solved as
+ * follows. Since PyObject-finalizers are based on
+ * {@link org.python.core.finalization.FinalizeTrigger}s, Jython has full control about
+ * these finalization process from a central point. Before such a finalizer runs, it calls
+ * {@link #notifyPreFinalization()} and when it is done, it calls
+ * {@link #notifyPostFinalization()}. While processing of a finalizer can be of arbitrary
+ * duration, it widely holds that Java's gc-thread calls the next finalizer almost
+ * instantaneously after the former. That means that a timestamp taken in
+ * {@link #notifyPreFinalization()} is usually delayed only few milliseconds
+ * - often even reported as 0 milliseconds - after the last taken timestamp in
+ * {@link #notifyPostFinalization()} (i.e. that was called by the previous finalizer).
+ * Jython's gc-module assumes the end of Java's finalization process if
+ * {@link #postFinalizationTimeOut} milliseconds passed after a call of
+ * {@link #notifyPostFinalization()} without another call to
+ * {@link #notifyPreFinalization()} in that time. The default value of
+ * {@link #postFinalizationTimeOut} is {@code 100}, which is far larger than the
+ * usual almost-zero duration between finalizer calls.<br>
+ * This process can be disturbed by third-party finalizers of non-PyObjects brought
+ * into the process by external libraries. If these finalizers are of short duration
+ * (which applies to typical finalizers), one can deal with this by adjusting
+ * {@link #postFinalizationTimeOut}, which was declared {@code public} for exactly this
+ * purpose. However if the external framework causing the issue is Jython-aware, a
+ * cleaner solution would be to let its finalizers call {@link #notifyPreFinalization()}
+ * and {@link #notifyPostFinalization()} appropriately. In that case these finalizers
+ * must not terminate by throwing an exception before {@link #notifyPostFinalization()}
+ * was called. This is a strict requirement, since a deadlock can be caused otherwise.<br>
+ * <br>
+ * Note that the management API
+ * (c.f.
+ * <a href="https://docs.oracle.com/javase/7/docs/jre/api/management/extension/index.html?com/sun/management/GcInfo.html
+ * target="_blank">com.sun.management.GarbageCollectionNotificationInfo</a>) does not emit any
+ * notifications that allow to detect the end of the finalization-phase. So this API
+ * provides no alternative to the described technique.
+ * </p>
+ * <p>
+ * Usually Java's gc provides hardly any guarantee about its collection-  and finalization-
+ * process. It not even guarantees that finalizers are called at all (c.f.
+ * <a href="http://howtodoinjava.com/2012/10/31/why-not-to-use-finalize-method-in-java"
+ * target="_blank">http://howtodoinjava.com/2012/10/31/why-not-to-use-finalize-method-in-java</a>).
+ * While at least the most common JVM implementations usually <b>do</b> call finalizers
+ * reliably under normal conditions, there still is no specific finalization-order guaranteed
+ * (one might reasonably expect that this would be related to reference-connection graph
+ * topology, but this appears not to be the case).
+ * However Jython now offers some functionality to compensate this
+ * situation. Via {@link #registerPreFinalizationProcess(Runnable)} and
+ * {@link #registerPostFinalizationProcess(Runnable)} and related methods one can now
+ * listen to beginning and end of the finalization process. Note that this functionality
+ * relies on the technique described in the former paragraph (i.e. based on calls to
+ * {@link #notifyPreFinalization()} and {@link #notifyPostFinalization()}) and thus
+ * underlies its unsafety, if third-party finalizers are involved. Such finalizers can
+ * cause false-positive runs of registered (pre/post)-finalization-processes, so this
+ * feature should be used with some care. It is recommended to use it only in such a way
+ * that false-positive runs would not cause serious harm, but only some loss in
+ * performance or so.
+ * </p>
+ */
 public class gc {
     /**
      * A constant that can occur as result of {@link #collect()} and
@@ -347,7 +488,7 @@
     private static List<Runnable> preFinalizationProcess, postFinalizationProcess;
     private static List<Runnable> preFinalizationProcessRemove, postFinalizationProcessRemove;
     private static Thread postFinalizationProcessor;
-    private static long postFinalizationTimeOut = 100;
+    public static long postFinalizationTimeOut = 100;
     private static long postFinalizationTimestamp = System.currentTimeMillis()-2*postFinalizationTimeOut;
     private static int openFinalizeCount = 0;
     private static boolean postFinalizationPending = false;
@@ -611,8 +752,7 @@
         }
 
         protected void finalize() throws Throwable {
-            //TODO: Find out why this would cause test to fail:
-            //notifyPreFinalization();
+            notifyPreFinalization();
             if ((gcFlags & VERBOSE_COLLECT) != 0) {
                 writeDebug("gc", "Sentinel finalizer called...");
             }
@@ -631,7 +771,7 @@
             if ((gcFlags & VERBOSE_COLLECT) != 0) {
                 writeDebug("gc", "Sentinel finalizer done");
             }
-            //notifyPostFinalization();
+            notifyPostFinalization();
         }
     }
 
@@ -898,7 +1038,7 @@
 
 //--------------Finalization preprocess/postprocess section--------------------
 
-    protected static class PostFinalizationProcessor implements Runnable {
+    private static class PostFinalizationProcessor implements Runnable {
         protected static PostFinalizationProcessor defaultInstance =
                 new PostFinalizationProcessor();
 
@@ -1627,8 +1767,8 @@
                 // objects get monitored, because with monitorGlobal activated
                 // the objects get monitored just when they are created and some
                 // of them are in an invalid state then and cannot directly obtain
-                // their string representation (produce overflow errors and such bad
-                // stuff). So we do it here...
+                // their string representation (would produce overflow errors and
+                // such bad stuff). So we do it here...
                 List<WeakReferenceGC> lst = new ArrayList<>();
                 for (WeakReferenceGC wr: monitoredObjects) {
                     if (wr.str == null) {
@@ -1666,7 +1806,6 @@
             } else if ((gcFlags & VERBOSE_COLLECT) != 0) {
                 writeDebug("gc", "sync collect done.");
             }
-
             delayedFinalizationMode = DO_NOTHING_SPECIAL;
             gcRunning.set(false);
             result = stat[0];
@@ -1766,8 +1905,6 @@
             }
         }
         maxWaitTime = initWaitTime;
-        WeakReference<GCSentinel> sentRef =
-        new WeakReference<>(new GCSentinel(Thread.currentThread()), gcTrash);
         lastRemoveTimeStamp = System.currentTimeMillis();
         if (finalizeWaitCount != 0) {
             System.err.println("Finalize wait count should be initially 0!");
@@ -1812,56 +1949,31 @@
             writeDebug("gc", "call System.gc.");
         }
         cyclicLookup = null;
-        System.gc();
-        List<WeakReferenceGC> collectBuffer = null;
-        if (needsCollectBuffer()) {
-            collectBuffer = new ArrayList<>();
+        List<WeakReferenceGC> collectBuffer;
+        
+        /* The following out-commented block is a nice idea to sync gc in a more
+         * elegant way. Unfortunately it proved infeasible because MXBean appears
+         * to be no reliable measurement for gc to have finished enqueueing trash.
+         * We leave it here to document this failed approach for future generations,
+         * so nobody needs to waste time on this again/can quickly reproduce how
+         * it fails.
+
+        // Yes, Errors should not be caught, but in this case we have a very good
+        // reason for it.
+        // collectSyncViaMXBean uses inofficial API, i.e. classes from com.sun.management.
+        // In case that a JVM does not provide this API, we have a less elegant fallback
+        // at hand, which is based on a sentinel- and timeout-technique.
+        try {
+            collectBuffer = collectSyncViaMXBean(stat, cyclic);
+        } catch (NoClassDefFoundError ncdfe) {
+            collectBuffer = collectSyncViaSentinel(stat, cyclic);
         }
-        long removeTime;
-        try {
-            while(true) {
-                removeTime = System.currentTimeMillis()-lastRemoveTimeStamp;
-                if (removeTime > maxWaitTime) {
-                    maxWaitTime = removeTime;
-                }
-                lastRemoveTimeStamp = System.currentTimeMillis();
-                trash = gcTrash.remove(Math.max(gcRecallTime, maxWaitTime*defaultWaitFactor));
-                if (trash != null) {
-                    if (trash instanceof WeakReferenceGC) {
-                        synchronized(monitoredObjects) {
-                            monitoredObjects.remove(trash);
-                        }
-                        //We avoid counting jython-specific objects in order to
-                        //obtain CPython-comparable results.
-                        if (cyclic.contains(trash) && !((WeakReferenceGC) trash).cls.contains("Java")) {
-                            ++stat[0];
-                            if (collectBuffer != null) {
-                                collectBuffer.add((WeakReferenceGC) trash);
-                            }
-                            if ((gcFlags & VERBOSE_COLLECT) != 0) {
-                                writeDebug("gc", "Collected cyclic object: "+trash);
-                            }
-                        }
-                        if (((WeakReferenceGC) trash).hasFinalizer) {
-                            ++finalizeWaitCount;
-                            if ((gcFlags & VERBOSE_FINALIZE) != 0) {
-                                writeDebug("gc", "Collected finalizable object: "+trash);
-                                writeDebug("gc", "New finalizeWaitCount: "+finalizeWaitCount);
-                            }
-                        }
-                    } else if (trash == sentRef && (gcFlags & VERBOSE_COLLECT) != 0) {
-                        writeDebug("gc", "Sentinel collected.");
-                    }
-                } else {
-                    System.gc();
-                }
-            }
-        } catch (InterruptedException iex) {}
-
         if ((gcFlags & VERBOSE_COLLECT) != 0) {
             writeDebug("gc", "all objects from run enqueud in trash queue.");
             writeDebug("gc", "pending finalizers: "+finalizeWaitCount);
-        }
+        }*/
+
+        collectBuffer = collectSyncViaSentinel(stat, cyclic);
         //lockPostFinalization assures that postFinalization process
         //only runs once per syncCollect-call.
         lockPostFinalization = false;
@@ -1870,40 +1982,15 @@
             postFinalizationProcessor.interrupt();
         }
         waitingForFinalizers = true;
-        if (finalizeWaitCount != 0) {
-            if ((gcFlags & VERBOSE_COLLECT) != 0) {
-                writeDebug("gc", "waiting for "+finalizeWaitCount+
-                        " pending finalizers.");
-                if (finalizeWaitCount < 0) {
-                    //Maybe even throw exception here?
-                    Py.writeError("gc", "There should never be "+
-                            "less than zero pending finalizers!");
-                }
-            }
-            // It is important to have the while  *inside* the synchronized block.
-            // Otherwise the notify might come just between the check and the wait,
-            // causing an endless waiting.
-            synchronized(GCSentinel.class) {
-                while (finalizeWaitCount != 0) {
-                    try {
-                        GCSentinel.class.wait();
-                    } catch (InterruptedException ie2) {}
-                }
-            }
-            if ((gcFlags & VERBOSE_COLLECT) != 0) {
-                writeDebug("gc", "no more finalizers pending.");
-            }
-        }
+        waitForFinalizers();
         waitingForFinalizers = false;
-
         if (postFinalizationPending) {
             if ((gcFlags & VERBOSE_COLLECT) != 0) {
-                writeDebug("gc",
-                        "waiting for pending post-finalization process.");
+                writeDebug("gc", "waiting for pending post-finalization process.");
             }
             // It is important to have the while (which is actually an "if" since the
             // InterruptedException is very unlikely to occur) *inside* the synchronized
-            // block. Otherwise the notify might come just between the check and the wait,
+            // block. Otherwise the notification might come just between the check and the wait,
             // causing an endless waiting. This is no pure academic consideration, but was
             // actually observed to happen from time to time, especially on faster systems.
             synchronized(PostFinalizationProcessor.class) {
@@ -1992,6 +2079,247 @@
         stat[1] -= abortedCyclicFinalizers;
     }
 
+    /*
+     * The following out-commented section belongs to the out-commented
+     * block on MXBean-based GC sync somewhere above.
+     * We keep it here to document this failed approach and to enable
+     * future developers to quickly reproduce and analyse this failure.
+
+    private static class GCListener implements NotificationListener {
+        private int index;
+        private Thread toNotify;
+
+        public GCListener(int index, Thread toNotify) {
+            this.index = index;
+            this.toNotify = toNotify;
+        }
+
+        public void handleNotification(Notification notif, Object handback) {
+            if (waitForGCNotification[index]) {
+                String notifType = notif.getType();
+                if (notifType.equals(
+                        com.sun.management.GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION)) {
+                    // retrieve the garbage collection notification information
+                    CompositeData cd = (CompositeData) notif.getUserData();
+                    com.sun.management.GarbageCollectionNotificationInfo info =
+                            com.sun.management.GarbageCollectionNotificationInfo.from(cd);
+                    if (info.getGcCause().equals("System.gc()") &&
+                            info.getGcAction().startsWith("end of ") &&
+                            info.getGcAction().endsWith(" GC")) {
+                        synchronized (GCListener.class) {
+                            --outstandingNotifications;
+                            if (toNotify != null && waitingForTrash && outstandingNotifications == 0) {
+                                toNotify.interrupt();
+                                toNotify = null;
+                            }
+                        }
+                    }
+                }
+            }
+        }
+    }
+
+    private static boolean[] waitForGCNotification;
+    private static int outstandingNotifications = 0;
+    private static List<GarbageCollectorMXBean> currentGCBeans;
+    private static String failFast = null;
+    private static boolean waitingForTrash = false;
+    private static List<WeakReferenceGC> collectSyncViaMXBean(int[] stat, Set<WeakReferenceGC> cyclic) {
+        // This step should be done first in order to fail fast,
+        // if com.sun.management is not available.
+        if (failFast == null) {
+            failFast =
+                com.sun.management.GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION;
+        }
+        //Reaching this line successfully means that com.sun.management exists.
+        Reference<? extends Object> trash;
+        currentGCBeans = ManagementFactory.getGarbageCollectorMXBeans();
+        long[] initialGCCounts = new long[currentGCBeans.size()];
+        GCListener[] listeners = new GCListener[initialGCCounts.length];
+        int i = 0;
+        for (GarbageCollectorMXBean gcb: currentGCBeans) {
+            listeners[i] = new GCListener(i, Thread.currentThread());
+            ((NotificationBroadcaster) gcb).addNotificationListener(
+                    listeners[i], null, null);
+            initialGCCounts[i++] = gcb.getCollectionCount();
+        }
+        if (waitForGCNotification == null || waitForGCNotification.length != initialGCCounts.length) {
+            waitForGCNotification = new boolean[initialGCCounts.length];
+        }
+        synchronized (GCListener.class) {
+            boolean gcRunDetected = false;
+            outstandingNotifications = 0;
+            while (!gcRunDetected) {
+                System.gc();
+                for (i = 0; i < waitForGCNotification.length; ++i) {
+                    waitForGCNotification[i] =
+                            currentGCBeans.get(i).getCollectionCount() > initialGCCounts[i];
+                    ++outstandingNotifications;
+                    if (waitForGCNotification[i]) {
+                        // at least one counter should change if a gc-run occurred.
+                        gcRunDetected = true;
+                    }
+                }
+            }
+        }
+        List<WeakReferenceGC> collectBuffer = null;
+        if (needsCollectBuffer()) {
+            collectBuffer = new ArrayList<>();
+        }
+        while (outstandingNotifications > 0) {
+            try {
+                waitingForTrash = true;
+                Thread.sleep(2000);
+            } catch (InterruptedException ie)
+            {
+            }
+            waitingForTrash = false;
+        }
+        trash = gcTrash.poll();
+        while (trash != null) {
+            if (trash instanceof WeakReferenceGC) {
+                synchronized(monitoredObjects) {
+                    monitoredObjects.remove(trash);
+                }
+                //We avoid counting jython-specific objects in order to
+                //obtain CPython-comparable results.
+                if (cyclic.contains(trash) && !((WeakReferenceGC) trash).cls.contains("Java")) {
+                    ++stat[0];
+                    if (collectBuffer != null) {
+                        collectBuffer.add((WeakReferenceGC) trash);
+                    }
+                    if ((gcFlags & VERBOSE_COLLECT) != 0) {
+                        writeDebug("gc", "Collected cyclic object: "+trash);
+                    }
+                }
+                if (((WeakReferenceGC) trash).hasFinalizer) {
+                    ++finalizeWaitCount;
+                    if ((gcFlags & VERBOSE_FINALIZE) != 0) {
+                        writeDebug("gc", "Collected finalizable object: "+trash);
+                        writeDebug("gc", "New finalizeWaitCount: "+finalizeWaitCount);
+                    }
+                }
+            }
+            trash = gcTrash.poll();
+        }
+        try {
+            Thread.sleep(200);
+        } catch (InterruptedException ie) {
+        }
+        trash = gcTrash.poll();
+        if (trash != null) {
+            //This should not happen, but actually does.
+            //So MXBean-based sync is not reliable!
+            System.out.println("Late trash: "+trash);
+            System.out.println("Late trash: "+trash.getClass());
+            System.out.println("Late trash: "+System.identityHashCode(trash));
+            int count = 0;
+            while (trash != null) {
+                System.out.println("Late trash "+count+": "+trash);
+                ++count;
+                trash = gcTrash.poll();
+            }
+            System.out.println("Late trash count: "+count);
+        }
+        i = 0;
+        for (GarbageCollectorMXBean gcb: currentGCBeans) {
+            try {
+                ((NotificationBroadcaster) gcb).removeNotificationListener(
+                        listeners[i++]);
+            } catch (ListenerNotFoundException lnfe) {
+            }
+        }
+        return collectBuffer;
+    }
+    */
+
+    private static List<WeakReferenceGC> collectSyncViaSentinel(int[] stat, Set<WeakReferenceGC> cyclic) {
+        WeakReference<GCSentinel> sentRef =
+                new WeakReference<>(new GCSentinel(Thread.currentThread()), gcTrash);
+        Reference<? extends Object> trash;
+        System.gc();
+        List<WeakReferenceGC> collectBuffer = null;
+        if (needsCollectBuffer()) {
+            collectBuffer = new ArrayList<>();
+        }
+        long removeTime;
+        try {
+            while(true) {
+                removeTime = System.currentTimeMillis()-lastRemoveTimeStamp;
+                if (removeTime > maxWaitTime) {
+                    maxWaitTime = removeTime;
+                }
+                lastRemoveTimeStamp = System.currentTimeMillis();
+                trash = gcTrash.remove(Math.max(gcRecallTime, maxWaitTime*defaultWaitFactor));
+                if (trash != null) {
+                    if (trash instanceof WeakReferenceGC) {
+                        synchronized(monitoredObjects) {
+                            monitoredObjects.remove(trash);
+                        }
+                        //We avoid counting jython-specific objects in order to
+                        //obtain CPython-comparable results.
+                        if (cyclic.contains(trash) && !((WeakReferenceGC) trash).cls.contains("Java")) {
+                            ++stat[0];
+                            if (collectBuffer != null) {
+                                collectBuffer.add((WeakReferenceGC) trash);
+                            }
+                            if ((gcFlags & VERBOSE_COLLECT) != 0) {
+                                writeDebug("gc", "Collected cyclic object: "+trash);
+                            }
+                        }
+                        if (((WeakReferenceGC) trash).hasFinalizer) {
+                            ++finalizeWaitCount;
+                            if ((gcFlags & VERBOSE_FINALIZE) != 0) {
+                                writeDebug("gc", "Collected finalizable object: "+trash);
+                                writeDebug("gc", "New finalizeWaitCount: "+finalizeWaitCount);
+                            }
+                        }
+                    } else if (trash == sentRef && (gcFlags & VERBOSE_COLLECT) != 0) {
+                        writeDebug("gc", "Sentinel collected.");
+                    }
+                } else {
+                    System.gc();
+                }
+            }
+        } catch (InterruptedException iex) {}
+        return collectBuffer;
+    }
+
+    private static void waitForFinalizers() {
+        if (finalizeWaitCount != 0) {
+            if ((gcFlags & VERBOSE_COLLECT) != 0) {
+                writeDebug("gc", "waiting for "+finalizeWaitCount+
+                        " pending finalizers.");
+                if (finalizeWaitCount < 0) {
+                    //Maybe even throw exception here?
+                    Py.writeError("gc", "There should never be "+
+                            "less than zero pending finalizers!");
+                }
+            }
+            // It is important to have the while *inside* the synchronized block.
+            // Otherwise the notify might come just between the check and the wait,
+            // causing an endless waiting.
+            synchronized(GCSentinel.class) {
+                while (finalizeWaitCount != 0) {
+                    try {
+                        GCSentinel.class.wait();
+                    } catch (InterruptedException ie2) {}
+                }
+            }
+            if ((gcFlags & VERBOSE_COLLECT) != 0) {
+                writeDebug("gc", "no more finalizers pending.");
+            }
+        }
+        //Can the following block be skipped if monitor global is active?
+        //No, because there could already be unmonitored finalizable objects!
+        while (openFinalizeCount > 0 || System.currentTimeMillis() - postFinalizationTimestamp
+                < postFinalizationTimeOut) {
+            try {
+                Thread.sleep(postFinalizationTimeOut);
+            } catch (InterruptedException ie) {}
+        }
+    }
+
     /**
      * Not supported by Jython.
      * Throws {@link org.python.core.Py#NotImplementedError}.
@@ -2748,7 +3076,7 @@
 
 //--------------Visitproc section----------------------------------------------
 
-    static class ReferentsFinder implements Visitproc {
+    private static class ReferentsFinder implements Visitproc {
         public static ReferentsFinder defaultInstance = new ReferentsFinder();
 
         /**
@@ -2771,7 +3099,7 @@
      * knows they need to be explored further. Only traversable objects are
      * considered by this visitproc.
      */
-    static class ReachableFinder implements Visitproc {
+    private static class ReachableFinder implements Visitproc {
         public static ReachableFinder defaultInstance = new ReachableFinder();
 
         /**
@@ -2790,7 +3118,7 @@
         }
     }
 
-    static class ReachableFinderWeakRefs implements Visitproc {
+    private static class ReachableFinderWeakRefs implements Visitproc {
         public static ReachableFinderWeakRefs defaultInstance = new ReachableFinderWeakRefs();
 
         @SuppressWarnings("unchecked")
@@ -2808,7 +3136,7 @@
         }
     }
 
-    static class ReferrerFinder implements Visitproc {
+    private static class ReferrerFinder implements Visitproc {
         public static ReferrerFinder defaultInstance = new ReferrerFinder();
 
         /**
@@ -2831,7 +3159,7 @@
      * behavior. This method is useful if one is not interested in the referrers,
      * but only wants to know (quickly) whether a connection exists or not.
      */
-    static class RefersToSetFinder implements Visitproc {
+    private static class RefersToSetFinder implements Visitproc {
         public static RefersToSetFinder defaultInstance = new RefersToSetFinder();
 
         @SuppressWarnings("unchecked")
@@ -2855,7 +3183,7 @@
      * By repeated use one can collect all objects referring to a given set
      * of objects in another set.
      */
-    static class RefInListFinder implements Visitproc {
+    private static class RefInListFinder implements Visitproc {
         public static RefInListFinder defaultInstance = new RefInListFinder();
         public boolean found = false;
 
@@ -2876,7 +3204,7 @@
         }
     }
 
-    static class ObjectInListFinder implements Visitproc {
+    private static class ObjectInListFinder implements Visitproc {
         public static ObjectInListFinder defaultInstance = new ObjectInListFinder();
         public boolean found = false;
 
diff --git a/src/org/python/modules/zipimport/zipimporter.java b/src/org/python/modules/zipimport/zipimporter.java
--- a/src/org/python/modules/zipimport/zipimporter.java
+++ b/src/org/python/modules/zipimport/zipimporter.java
@@ -20,6 +20,8 @@
 import org.python.core.PySystemState;
 import org.python.core.PyTuple;
 import org.python.core.PyType;
+import org.python.core.Traverseproc;
+import org.python.core.Visitproc;
 import org.python.core.util.FileUtil;
 import org.python.core.util.StringUtil;
 import org.python.core.util.importer;
@@ -34,7 +36,7 @@
  * @author Philip Jenvey
  */
 @ExposedType(name = "zipimport.zipimporter", base = PyObject.class)
-public class zipimporter extends importer<PyObject> {
+public class zipimporter extends importer<PyObject> implements Traverseproc {
 
     public static final PyType TYPE = PyType.fromClass(zipimporter.class);
 
@@ -547,4 +549,22 @@
             }
         }
     }
+
+
+    /* Traverseproc implementation */
+    @Override
+    public int traverse(Visitproc visit, Object arg) {
+    	if (files != null) {
+    		int retVal = visit.visit(files, arg);
+    		if (retVal != 0) {
+    			return retVal;
+    		}
+    	}
+        return sys == null ? 0 : visit.visit(sys, arg);
+    }
+
+    @Override
+    public boolean refersDirectlyTo(PyObject ob) {
+        return ob != null && (ob == files || ob == sys);
+    }
 }
diff --git a/tests/java/javatests/GCTestHelper.java b/tests/java/javatests/GCTestHelper.java
--- a/tests/java/javatests/GCTestHelper.java
+++ b/tests/java/javatests/GCTestHelper.java
@@ -21,7 +21,7 @@
     }
 
     /**
-     * In contrast to Nasty finalizer, this class - still equally
+     * In contrast to NastyFinalizer, this class - still equally
      * time-consuming - calls {@code gc.notifyPreFinalization()}
      * and {@code gc.notifyPostFinalization()} and thus lets all
      * tests work as expected.

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 13 19:42:37 2015
From: jython-checkins at python.org (jim.baker)
Date: Fri, 13 Mar 2015 18:42:37 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Mark_PyUnicode=2EIndexTrans?=
 =?utf-8?q?lator_interface_as_Serializable?=
Message-ID: <20150313184236.23872.96392@psf.io>

https://hg.python.org/jython/rev/d8be9df6530c
changeset:   7610:d8be9df6530c
user:        Eliya Sadan <eliyasadan at gmail.com>
date:        Fri Mar 13 12:42:08 2015 -0600
summary:
  Mark PyUnicode.IndexTranslator interface as Serializable

Fixes http://bugs.jython.org/issue2289

files:
  ACKNOWLEDGMENTS                    |  1 +
  src/org/python/core/PyUnicode.java |  3 ++-
  2 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/ACKNOWLEDGMENTS b/ACKNOWLEDGMENTS
--- a/ACKNOWLEDGMENTS
+++ b/ACKNOWLEDGMENTS
@@ -150,6 +150,7 @@
     Jiwon Seo
     Dieter Vandenbussche
     Paolo Dina
+    Eliya Sadan
 
 Local Variables:
 mode: indented-text
diff --git a/src/org/python/core/PyUnicode.java b/src/org/python/core/PyUnicode.java
--- a/src/org/python/core/PyUnicode.java
+++ b/src/org/python/core/PyUnicode.java
@@ -1,5 +1,6 @@
 package org.python.core;
 
+import java.io.Serializable;
 import java.util.Collection;
 import java.util.Collections;
 import java.util.Iterator;
@@ -140,7 +141,7 @@
      * Index translation between code point index (as seen by Python) and UTF-16 index (as used in
      * the Java String.
      */
-    private interface IndexTranslator {
+    private interface IndexTranslator extends Serializable {
 
         /** Number of supplementary characters (hence point code length may be found). */
         public int suppCount();

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 13 19:42:37 2015
From: jython-checkins at python.org (jim.baker)
Date: Fri, 13 Mar 2015 18:42:37 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Add_JUnit_tests_to_verify_r?=
 =?utf-8?q?oundtrip_serialization_of_core_types?=
Message-ID: <20150313184236.13205.84657@psf.io>

https://hg.python.org/jython/rev/8e85dc69027c
changeset:   7609:8e85dc69027c
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 13 12:39:20 2015 -0600
summary:
  Add JUnit tests to verify roundtrip serialization of core types

Includes verification of PyUnicode serialization support, as need for
the fix for http://bugs.jython.org/issue2289

files:
  tests/java/org/python/tests/SerializationTest.java |  102 ++++++++++
  1 files changed, 102 insertions(+), 0 deletions(-)


diff --git a/tests/java/org/python/tests/SerializationTest.java b/tests/java/org/python/tests/SerializationTest.java
--- a/tests/java/org/python/tests/SerializationTest.java
+++ b/tests/java/org/python/tests/SerializationTest.java
@@ -2,13 +2,34 @@
 
 import java.io.ByteArrayInputStream;
 import java.io.ByteArrayOutputStream;
+import java.io.InputStream;
+import java.io.InvalidClassException;
 import java.io.IOException;
+import java.io.ObjectInputStream;
+import java.io.ObjectStreamClass;
+import java.io.OutputStream;
 import java.io.ObjectOutputStream;
+import java.io.OutputStream;
+import java.lang.AssertionError;
+import java.lang.Integer;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.Map;
+import java.util.Queue;
 
 import junit.framework.TestCase;
+import static org.junit.Assert.*;
 
+import org.python.core.Py;
+import org.python.core.PyDictionary;
+import org.python.core.PyFrozenSet;
+import org.python.core.PyList;
+import org.python.core.PyObject;
+import org.python.core.PySet;
 import org.python.core.PyStringMap;
 import org.python.core.PySystemState;
+import org.python.core.PyTuple;
+import org.python.modules._csv.PyDialect;
 import org.python.util.PythonInterpreter;
 import org.python.util.PythonObjectInputStream;
 
@@ -34,4 +55,85 @@
         interp.set("t", this);
         interp.exec("t.testDirect()");
     }
+
+    public void testBasicTypes() {
+        assertRoundtrip(Py.None);
+        assertRoundtrip(Py.True);
+        assertRoundtrip(Py.False);
+        assertRoundtrip(Py.newInteger(42));
+        assertRoundtrip(Py.newLong(47));
+        assertRoundtrip(Py.newString("Jython: Python for the Java Platform"));
+        assertRoundtrip(Py.newUnicode("Drink options include \uD83C\uDF7A, \uD83C\uDF75, \uD83C\uDF77, and \u2615"));
+        Map<PyObject, PyObject> map = new HashMap<>();
+        map.put(Py.newString("OEIS interesting number"), Py.newInteger(14228));
+        map.put(Py.newString("Hardy-Ramanujan number"), Py.newInteger(1729));
+        assertRoundtrip(new PyDictionary(map));
+        assertRoundtrip(new PyList(new PyObject[]{Py.newInteger(1), Py.newInteger(28), Py.newInteger(546), Py.newInteger(9450), Py.newInteger(157773)})); // A001234
+        assertRoundtrip(new PySet(new PyObject[]{Py.Zero, Py.One}));
+        assertRoundtrip(new PyFrozenSet(new PyTuple(Py.newInteger(1), Py.newInteger(2), Py.newInteger(3))));
+        assertRoundtrip(new PyTuple(Py.newInteger(2), Py.newInteger(8), Py.newInteger(248), Py.newInteger(113281))); // A012345
+    }
+
+    private static class CloneOutput extends ObjectOutputStream {
+        Queue<Class<?>> classQueue = new LinkedList<Class<?>>();
+
+        CloneOutput(OutputStream out) throws IOException {
+            super(out);
+        }
+
+        @Override
+        protected void annotateClass(Class<?> c) {
+            classQueue.add(c);
+        }
+
+        @Override
+        protected void annotateProxyClass(Class<?> c) {
+            classQueue.add(c);
+        }
+    }
+
+    private static class CloneInput extends ObjectInputStream {
+        private final CloneOutput output;
+
+        CloneInput(InputStream in, CloneOutput output) throws IOException {
+            super(in);
+            this.output = output;
+        }
+
+        @Override
+        protected Class<?> resolveClass(ObjectStreamClass osc)
+                throws IOException, ClassNotFoundException {
+            Class<?> c = output.classQueue.poll();
+            String expected = osc.getName();
+            String found = (c == null) ? null : c.getName();
+            if (!expected.equals(found)) {
+                throw new InvalidClassException("Classes desynchronized: " +
+                        "found " + found + " when expecting " + expected);
+            }
+            return c;
+        }
+
+        @Override
+        protected Class<?> resolveProxyClass(String[] interfaceNames)
+                throws IOException, ClassNotFoundException {
+            return output.classQueue.poll();
+        }
+    }
+
+    public void assertRoundtrip(Object obj) {
+        try {
+            ByteArrayOutputStream output = new ByteArrayOutputStream();
+            CloneOutput serializer = new CloneOutput(output);
+            serializer.writeObject(obj);
+            serializer.close();
+            ByteArrayInputStream input = new ByteArrayInputStream(output.toByteArray());
+            CloneInput unserializer = new CloneInput(input, serializer);
+            assertEquals(obj, unserializer.readObject());
+        } catch (IOException ioe) {
+            throw new AssertionError(ioe);
+        } catch (ClassNotFoundException ex) {
+            throw new AssertionError(ex);
+        }
+    }
 }
+

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 13 22:24:47 2015
From: jython-checkins at python.org (jim.baker)
Date: Fri, 13 Mar 2015 21:24:47 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Remove_Jython-specific_pyth?=
 =?utf-8?q?onpath=2C_test=5Fpythonpath=3B_update_pwd_to_support?=
Message-ID: <20150313212447.23868.61759@psf.io>

https://hg.python.org/jython/rev/8c1a3b72913e
changeset:   7611:8c1a3b72913e
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 13 15:24:16 2015 -0600
summary:
  Remove Jython-specific pythonpath, test_pythonpath; update pwd to support unicode

Fixes http://bugs.jython.org/issue2274

files:
  CPythonLib.includes        |    1 +
  Lib/posixpath.py           |  513 ------------------------
  Lib/pwd.py                 |    8 +-
  Lib/test/test_posixpath.py |  521 -------------------------
  4 files changed, 5 insertions(+), 1038 deletions(-)


diff --git a/CPythonLib.includes b/CPythonLib.includes
--- a/CPythonLib.includes
+++ b/CPythonLib.includes
@@ -116,6 +116,7 @@
 plistlib.py
 poplib.py
 posixfile.py
+posixpath.py
 pprint.py
 profile.py
 pstats.py
diff --git a/Lib/posixpath.py b/Lib/posixpath.py
deleted file mode 100644
--- a/Lib/posixpath.py
+++ /dev/null
@@ -1,513 +0,0 @@
-"""Common operations on Posix pathnames.
-
-Instead of importing this module directly, import os and refer to
-this module as os.path.  The "os.path" name is an alias for this
-module on Posix systems; on other systems (e.g. Mac, Windows),
-os.path provides the same operations in a manner specific to that
-platform, and is an alias to another module (e.g. macpath, ntpath).
-
-Some of this can actually be useful on non-Posix systems too, e.g.
-for manipulation of the pathname component of URLs.
-"""
-
-import os
-import stat
-
-__all__ = ["normcase","isabs","join","splitdrive","split","splitext",
-           "basename","dirname","commonprefix","getsize","getmtime",
-           "getatime","getctime","islink","exists","lexists","isdir","isfile",
-           "walk","expanduser","expandvars","normpath","abspath",
-           "samefile",
-           "curdir","pardir","sep","pathsep","defpath","altsep","extsep",
-           "devnull","realpath","supports_unicode_filenames", "relpath"]
-
-# strings representing various path-related bits and pieces
-curdir = '.'
-pardir = '..'
-extsep = '.'
-sep = '/'
-pathsep = ':'
-defpath = ':/bin:/usr/bin'
-altsep = None
-devnull = '/dev/null'
-
-# Normalize the case of a pathname.  Trivial in Posix, string.lower on Mac.
-# On MS-DOS this may also turn slashes into backslashes; however, other
-# normalizations (such as optimizing '../' away) are not allowed
-# (another function should be defined to do that).
-
-def normcase(s):
-    """Normalize case of pathname.  Has no effect under Posix"""
-    return s
-
-
-# Return whether a path is absolute.
-# Trivial in Posix, harder on the Mac or MS-DOS.
-
-def isabs(s):
-    """Test whether a path is absolute"""
-    return s.startswith('/')
-
-
-# Join pathnames.
-# Ignore the previous parts if a part is absolute.
-# Insert a '/' unless the first part is empty or already ends in '/'.
-
-def join(a, *p):
-    """Join two or more pathname components, inserting '/' as needed"""
-    path = a
-    for b in p:
-        if b.startswith('/'):
-            path = b
-        elif path == '' or path.endswith('/'):
-            path +=  b
-        else:
-            path += '/' + b
-    return path
-
-
-# Split a path in head (everything up to the last '/') and tail (the
-# rest).  If the path ends in '/', tail will be empty.  If there is no
-# '/' in the path, head  will be empty.
-# Trailing '/'es are stripped from head unless it is the root.
-
-def split(p):
-    """Split a pathname.  Returns tuple "(head, tail)" where "tail" is
-    everything after the final slash.  Either part may be empty."""
-    i = p.rfind('/') + 1
-    head, tail = p[:i], p[i:]
-    if head and head != '/'*len(head):
-        head = head.rstrip('/')
-    return head, tail
-
-
-# Split a path in root and extension.
-# The extension is everything starting at the last dot in the last
-# pathname component; the root is everything before that.
-# It is always true that root + ext == p.
-
-def splitext(p):
-    """Split the extension from a pathname.  Extension is everything from the
-    last dot to the end.  Returns "(root, ext)", either part may be empty."""
-    i = p.rfind('.')
-    if i<=p.rfind('/'):
-        return p, ''
-    else:
-        return p[:i], p[i:]
-
-
-# Split a pathname into a drive specification and the rest of the
-# path.  Useful on DOS/Windows/NT; on Unix, the drive is always empty.
-
-def splitdrive(p):
-    """Split a pathname into drive and path. On Posix, drive is always
-    empty."""
-    return '', p
-
-
-# Return the tail (basename) part of a path.
-
-def basename(p):
-    """Returns the final component of a pathname"""
-    return split(p)[1]
-
-
-# Return the head (dirname) part of a path.
-
-def dirname(p):
-    """Returns the directory component of a pathname"""
-    return split(p)[0]
-
-
-# Return the longest prefix of all list elements.
-
-def commonprefix(m):
-    "Given a list of pathnames, returns the longest common leading component"
-    if not m: return ''
-    s1 = min(m)
-    s2 = max(m)
-    n = min(len(s1), len(s2))
-    for i in xrange(n):
-        if s1[i] != s2[i]:
-            return s1[:i]
-    return s1[:n]
-
-# Get size, mtime, atime of files.
-
-def getsize(filename):
-    """Return the size of a file, reported by os.stat()."""
-    return os.stat(filename).st_size
-
-def getmtime(filename):
-    """Return the last modification time of a file, reported by os.stat()."""
-    return os.stat(filename).st_mtime
-
-def getatime(filename):
-    """Return the last access time of a file, reported by os.stat()."""
-    return os.stat(filename).st_atime
-
-def getctime(filename):
-    """Return the metadata change time of a file, reported by os.stat()."""
-    return os.stat(filename).st_ctime
-
-# Is a path a symbolic link?
-# This will always return false on systems where os.lstat doesn't exist.
-
-def islink(path):
-    """Test whether a path is a symbolic link"""
-    try:
-        st = os.lstat(path)
-    except (os.error, AttributeError):
-        return False
-    return stat.S_ISLNK(st.st_mode)
-
-
-# Does a path exist?
-# This is false for dangling symbolic links.
-
-def exists(path):
-    """Test whether a path exists.  Returns False for broken symbolic links"""
-    try:
-        st = os.stat(path)
-    except os.error:
-        return False
-    return True
-
-
-# Being true for dangling symbolic links is also useful.
-
-def lexists(path):
-    """Test whether a path exists.  Returns True for broken symbolic links"""
-    try:
-        st = os.lstat(path)
-    except os.error:
-        return False
-    return True
-
-
-# Is a path a directory?
-# This follows symbolic links, so both islink() and isdir() can be true
-# for the same path.
-
-def isdir(path):
-    """Test whether a path is a directory"""
-    try:
-        st = os.stat(path)
-    except os.error:
-        return False
-    return stat.S_ISDIR(st.st_mode)
-
-
-# Is a path a regular file?
-# This follows symbolic links, so both islink() and isfile() can be true
-# for the same path.
-
-def isfile(path):
-    """Test whether a path is a regular file"""
-    try:
-        st = os.stat(path)
-    except os.error:
-        return False
-    return stat.S_ISREG(st.st_mode)
-
-
-# Are two filenames really pointing to the same file?
-
-if not os._native_posix:
-    import java.io.File
-    import java.io.IOException
-    from org.python.core.Py import newString
-
-    def samefile(f1, f2):
-        """Test whether two pathnames reference the same actual file"""
-        canon1 = newString(java.io.File(_ensure_str(f1)).getCanonicalPath())
-        canon2 = newString(java.io.File(_ensure_str(f2)).getCanonicalPath())
-        return canon1 == canon2
-else:
-    def samefile(f1, f2):
-        """Test whether two pathnames reference the same actual file"""
-        s1 = os.stat(f1)
-        s2 = os.stat(f2)
-        return samestat(s1, s2)
-
-
-# XXX: Jython currently lacks fstat
-if hasattr(os, 'fstat'):
-    # Are two open files really referencing the same file?
-    # (Not necessarily the same file descriptor!)
-
-    def sameopenfile(fp1, fp2):
-        """Test whether two open file objects reference the same file"""
-        s1 = os.fstat(fp1)
-        s2 = os.fstat(fp2)
-        return samestat(s1, s2)
-
-    __all__.append("sameopenfile")
-
-
-# XXX: Pure Java stat lacks st_ino/st_dev
-if os._native_posix:
-    # Are two stat buffers (obtained from stat, fstat or lstat)
-    # describing the same file?
-
-    def samestat(s1, s2):
-        """Test whether two stat buffers reference the same file"""
-        return s1.st_ino == s2.st_ino and \
-               s1.st_dev == s2.st_dev
-
-
-    # Is a path a mount point?
-    # (Does this work for all UNIXes?  Is it even guaranteed to work by Posix?)
-
-    def ismount(path):
-        """Test whether a path is a mount point"""
-        try:
-            s1 = os.lstat(path)
-            s2 = os.lstat(join(path, '..'))
-        except os.error:
-            return False # It doesn't exist -- so not a mount point :-)
-        dev1 = s1.st_dev
-        dev2 = s2.st_dev
-        if dev1 != dev2:
-            return True     # path/.. on a different device as path
-        ino1 = s1.st_ino
-        ino2 = s2.st_ino
-        if ino1 == ino2:
-            return True     # path/.. is the same i-node as path
-        return False
-
-    __all__.extend(["samestat", "ismount"])
-
-
-# Directory tree walk.
-# For each directory under top (including top itself, but excluding
-# '.' and '..'), func(arg, dirname, filenames) is called, where
-# dirname is the name of the directory and filenames is the list
-# of files (and subdirectories etc.) in the directory.
-# The func may modify the filenames list, to implement a filter,
-# or to impose a different order of visiting.
-
-def walk(top, func, arg):
-    """Directory tree walk with callback function.
-
-    For each directory in the directory tree rooted at top (including top
-    itself, but excluding '.' and '..'), call func(arg, dirname, fnames).
-    dirname is the name of the directory, and fnames a list of the names of
-    the files and subdirectories in dirname (excluding '.' and '..').  func
-    may modify the fnames list in-place (e.g. via del or slice assignment),
-    and walk will only recurse into the subdirectories whose names remain in
-    fnames; this can be used to implement a filter, or to impose a specific
-    order of visiting.  No semantics are defined for, or required of, arg,
-    beyond that arg is always passed to func.  It can be used, e.g., to pass
-    a filename pattern, or a mutable object designed to accumulate
-    statistics.  Passing None for arg is common."""
-
-    try:
-        names = os.listdir(top)
-    except os.error:
-        return
-    func(arg, top, names)
-    for name in names:
-        name = join(top, name)
-        try:
-            st = os.lstat(name)
-        except os.error:
-            continue
-        if stat.S_ISDIR(st.st_mode):
-            walk(name, func, arg)
-
-
-# Expand paths beginning with '~' or '~user'.
-# '~' means $HOME; '~user' means that user's home directory.
-# If the path doesn't begin with '~', or if the user or $HOME is unknown,
-# the path is returned unchanged (leaving error reporting to whatever
-# function is called with the expanded path as argument).
-# See also module 'glob' for expansion of *, ? and [...] in pathnames.
-# (A function should also be defined to do full *sh-style environment
-# variable expansion.)
-
-def expanduser(path):
-    """Expand ~ and ~user constructions.  If user or $HOME is unknown,
-    do nothing."""
-    if not path.startswith('~'):
-        return path
-    i = path.find('/', 1)
-    if i < 0:
-        i = len(path)
-    if i == 1:
-        if 'HOME' not in os.environ:
-            return path
-        else:
-            userhome = os.environ['HOME']
-    else:
-        # XXX: Jython lacks the pwd module: '~user' isn't supported
-        return path
-    userhome = userhome.rstrip('/')
-    return userhome + path[i:]
-
-
-# Expand paths containing shell variable substitutions.
-# This expands the forms $variable and ${variable} only.
-# Non-existent variables are left unchanged.
-
-_varprog = None
-
-def expandvars(path):
-    """Expand shell variables of form $var and ${var}.  Unknown variables
-    are left unchanged."""
-    global _varprog
-    if '$' not in path:
-        return path
-    if not _varprog:
-        import re
-        _varprog = re.compile(r'\$(\w+|\{[^}]*\})')
-    i = 0
-    while True:
-        m = _varprog.search(path, i)
-        if not m:
-            break
-        i, j = m.span(0)
-        name = m.group(1)
-        if name.startswith('{') and name.endswith('}'):
-            name = name[1:-1]
-        if name in os.environ:
-            tail = path[j:]
-            path = path[:i] + os.environ[name]
-            i = len(path)
-            path += tail
-        else:
-            i = j
-    return path
-
-
-# Normalize a path, e.g. A//B, A/./B and A/foo/../B all become A/B.
-# It should be understood that this may change the meaning of the path
-# if it contains symbolic links!
-
-def normpath(path):
-    """Normalize path, eliminating double slashes, etc."""
-    if path == '':
-        return '.'
-    initial_slashes = path.startswith('/')
-    # POSIX allows one or two initial slashes, but treats three or more
-    # as single slash.
-    if (initial_slashes and
-        path.startswith('//') and not path.startswith('///')):
-        initial_slashes = 2
-    comps = path.split('/')
-    new_comps = []
-    for comp in comps:
-        if comp in ('', '.'):
-            continue
-        if (comp != '..' or (not initial_slashes and not new_comps) or
-             (new_comps and new_comps[-1] == '..')):
-            new_comps.append(comp)
-        elif new_comps:
-            new_comps.pop()
-    comps = new_comps
-    path = '/'.join(comps)
-    if initial_slashes:
-        path = '/'*initial_slashes + path
-    return path or '.'
-
-
-def abspath(path):
-    """Return an absolute path."""
-    if not isabs(path):
-        path = join(os.getcwd(), path)
-    return normpath(path)
-
-
-# Return a canonical path (i.e. the absolute location of a file on the
-# filesystem).
-
-def realpath(filename):
-    """Return the canonical path of the specified filename, eliminating any
-symbolic links encountered in the path."""
-    if isabs(filename):
-        bits = ['/'] + filename.split('/')[1:]
-    else:
-        bits = [''] + filename.split('/')
-
-    for i in range(2, len(bits)+1):
-        component = join(*bits[0:i])
-        # Resolve symbolic links.
-        if islink(component):
-            resolved = _resolve_link(component)
-            if resolved is None:
-                # Infinite loop -- return original component + rest of the path
-                return abspath(join(*([component] + bits[i:])))
-            else:
-                newpath = join(*([resolved] + bits[i:]))
-                return realpath(newpath)
-
-    return abspath(filename)
-
-
-if not os._native_posix:
-    def _resolve_link(path):
-        """Internal helper function.  Takes a path and follows symlinks
-        until we either arrive at something that isn't a symlink, or
-        encounter a path we've seen before (meaning that there's a loop).
-        """
-        try:
-            return newString(java.io.File(abspath(path)).getCanonicalPath())
-        except java.io.IOException:
-            return None
-else:
-    def _resolve_link(path):
-        """Internal helper function.  Takes a path and follows symlinks
-        until we either arrive at something that isn't a symlink, or
-        encounter a path we've seen before (meaning that there's a loop).
-        """
-        paths_seen = []
-        while islink(path):
-            if path in paths_seen:
-                # Already seen this path, so we must have a symlink loop
-                return None
-            paths_seen.append(path)
-            # Resolve where the link points to
-            resolved = os.readlink(path)
-            if not isabs(resolved):
-                dir = dirname(path)
-                path = normpath(join(dir, resolved))
-            else:
-                path = normpath(resolved)
-        return path
-
-def relpath(path, start=curdir):
-    """Return a relative version of a path"""
-
-    if not path:
-        raise ValueError("no path specified")
-
-    start_list = [x for x in abspath(start).split(sep) if x]
-    path_list = [x for x in abspath(path).split(sep) if x]
-
-    # Work out how much of the filepath is shared by start and path.
-    i = len(commonprefix([start_list, path_list]))
-
-    rel_list = [pardir] * (len(start_list)-i) + path_list[i:]
-    if not rel_list:
-        return curdir
-    return join(*rel_list)
-def _ensure_str(obj):
-    """Ensure obj is a string, otherwise raise a TypeError"""
-    if isinstance(obj, basestring):
-        return obj
-    raise TypeError('coercing to Unicode: need string or buffer, %s found' % \
-                        _type_name(obj))
-
-
-def _type_name(obj):
-    """Determine the appropriate type name of obj for display"""
-    TPFLAGS_HEAPTYPE = 1 << 9
-    type_name = ''
-    obj_type = type(obj)
-    is_heap = obj_type.__flags__ & TPFLAGS_HEAPTYPE == TPFLAGS_HEAPTYPE
-    if not is_heap and obj_type.__module__ != '__builtin__':
-        type_name = '%s.' % obj_type.__module__
-    type_name += obj_type.__name__
-    return type_name
-
-supports_unicode_filenames = False
diff --git a/Lib/pwd.py b/Lib/pwd.py
--- a/Lib/pwd.py
+++ b/Lib/pwd.py
@@ -11,7 +11,7 @@
 __all__ = ['getpwuid', 'getpwnam', 'getpwall']
 
 from os import _name, _posix_impl
-from org.python.core.Py import newString
+from org.python.core.Py import newStringOrUnicode
 import sys
 
 if _name == 'nt':
@@ -30,9 +30,9 @@
              'pw_dir', 'pw_shell']
 
     def __new__(cls, pwd):
-        pwd = (newString(pwd.loginName), newString(pwd.password), int(pwd.UID),
-               int(pwd.GID), newString(pwd.GECOS), newString(pwd.home),
-               newString(pwd.shell))
+        pwd = (newStringOrUnicode(pwd.loginName), newStringOrUnicode(pwd.password), int(pwd.UID),
+               int(pwd.GID), newStringOrUnicode(pwd.GECOS), newStringOrUnicode(pwd.home),
+               newStringOrUnicode(pwd.shell))
         return tuple.__new__(cls, pwd)
 
     def __getattr__(self, attr):
diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py
deleted file mode 100644
--- a/Lib/test/test_posixpath.py
+++ /dev/null
@@ -1,521 +0,0 @@
-import unittest
-from test import test_support
-
-import posixpath, os
-from posixpath import realpath, abspath, join, dirname, basename
-
-# An absolute path to a temporary filename for testing. We can't rely on TESTFN
-# being an absolute path, so we need this.
-
-ABSTFN = abspath(test_support.TESTFN)
-
-class PosixPathTest(unittest.TestCase):
-
-    def assertIs(self, a, b):
-        self.assert_(a is b)
-
-    def test_normcase(self):
-        # Check that normcase() is idempotent
-        p = "FoO/./BaR"
-        p = posixpath.normcase(p)
-        self.assertEqual(p, posixpath.normcase(p))
-
-        self.assertRaises(TypeError, posixpath.normcase)
-
-    def test_join(self):
-        self.assertEqual(posixpath.join("/foo", "bar", "/bar", "baz"), "/bar/baz")
-        self.assertEqual(posixpath.join("/foo", "bar", "baz"), "/foo/bar/baz")
-        self.assertEqual(posixpath.join("/foo/", "bar/", "baz/"), "/foo/bar/baz/")
-
-        self.assertRaises(TypeError, posixpath.join)
-
-    def test_splitdrive(self):
-        self.assertEqual(posixpath.splitdrive("/foo/bar"), ("", "/foo/bar"))
-
-        self.assertRaises(TypeError, posixpath.splitdrive)
-
-    def test_split(self):
-        self.assertEqual(posixpath.split("/foo/bar"), ("/foo", "bar"))
-        self.assertEqual(posixpath.split("/"), ("/", ""))
-        self.assertEqual(posixpath.split("foo"), ("", "foo"))
-        self.assertEqual(posixpath.split("////foo"), ("////", "foo"))
-        self.assertEqual(posixpath.split("//foo//bar"), ("//foo", "bar"))
-
-        self.assertRaises(TypeError, posixpath.split)
-
-    def test_splitext(self):
-        self.assertEqual(posixpath.splitext("foo.ext"), ("foo", ".ext"))
-        self.assertEqual(posixpath.splitext("/foo/foo.ext"), ("/foo/foo", ".ext"))
-        self.assertEqual(posixpath.splitext(".ext"), ("", ".ext"))
-        self.assertEqual(posixpath.splitext("/foo.ext/foo"), ("/foo.ext/foo", ""))
-        self.assertEqual(posixpath.splitext("foo.ext/"), ("foo.ext/", ""))
-        self.assertEqual(posixpath.splitext(""), ("", ""))
-        self.assertEqual(posixpath.splitext("foo.bar.ext"), ("foo.bar", ".ext"))
-
-        self.assertRaises(TypeError, posixpath.splitext)
-
-    def test_isabs(self):
-        self.assertIs(posixpath.isabs(""), False)
-        self.assertIs(posixpath.isabs("/"), True)
-        self.assertIs(posixpath.isabs("/foo"), True)
-        self.assertIs(posixpath.isabs("/foo/bar"), True)
-        self.assertIs(posixpath.isabs("foo/bar"), False)
-
-        self.assertRaises(TypeError, posixpath.isabs)
-
-    def test_splitdrive(self):
-        self.assertEqual(posixpath.splitdrive("/foo/bar"), ("", "/foo/bar"))
-
-        self.assertRaises(TypeError, posixpath.splitdrive)
-
-    def test_basename(self):
-        self.assertEqual(posixpath.basename("/foo/bar"), "bar")
-        self.assertEqual(posixpath.basename("/"), "")
-        self.assertEqual(posixpath.basename("foo"), "foo")
-        self.assertEqual(posixpath.basename("////foo"), "foo")
-        self.assertEqual(posixpath.basename("//foo//bar"), "bar")
-
-        self.assertRaises(TypeError, posixpath.basename)
-
-    def test_dirname(self):
-        self.assertEqual(posixpath.dirname("/foo/bar"), "/foo")
-        self.assertEqual(posixpath.dirname("/"), "/")
-        self.assertEqual(posixpath.dirname("foo"), "")
-        self.assertEqual(posixpath.dirname("////foo"), "////")
-        self.assertEqual(posixpath.dirname("//foo//bar"), "//foo")
-
-        self.assertRaises(TypeError, posixpath.dirname)
-
-    def test_commonprefix(self):
-        self.assertEqual(
-            posixpath.commonprefix([]),
-            ""
-        )
-        self.assertEqual(
-            posixpath.commonprefix(["/home/swenson/spam", "/home/swen/spam"]),
-            "/home/swen"
-        )
-        self.assertEqual(
-            posixpath.commonprefix(["/home/swen/spam", "/home/swen/eggs"]),
-            "/home/swen/"
-        )
-        self.assertEqual(
-            posixpath.commonprefix(["/home/swen/spam", "/home/swen/spam"]),
-            "/home/swen/spam"
-        )
-
-    def test_getsize(self):
-        f = open(test_support.TESTFN, "wb")
-        try:
-            f.write("foo")
-            f.close()
-            self.assertEqual(posixpath.getsize(test_support.TESTFN), 3)
-        finally:
-            if not f.closed:
-                f.close()
-            os.remove(test_support.TESTFN)
-
-    def test_time(self):
-        f = open(test_support.TESTFN, "wb")
-        try:
-            f.write("foo")
-            f.close()
-            f = open(test_support.TESTFN, "ab")
-            f.write("bar")
-            f.close()
-            f = open(test_support.TESTFN, "rb")
-            d = f.read()
-            f.close()
-            self.assertEqual(d, "foobar")
-
-            self.assert_(
-                posixpath.getctime(test_support.TESTFN) <=
-                posixpath.getmtime(test_support.TESTFN)
-            )
-        finally:
-            if not f.closed:
-                f.close()
-            os.remove(test_support.TESTFN)
-
-    @unittest.skip("FIXME: broken")
-    def test_islink(self):
-        self.assertIs(posixpath.islink(test_support.TESTFN + "1"), False)
-        f = open(test_support.TESTFN + "1", "wb")
-        try:
-            f.write("foo")
-            f.close()
-            self.assertIs(posixpath.islink(test_support.TESTFN + "1"), False)
-            if hasattr(os, "symlink"):
-                os.symlink(test_support.TESTFN + "1", test_support.TESTFN + "2")
-                self.assertIs(posixpath.islink(test_support.TESTFN + "2"), True)
-                os.remove(test_support.TESTFN + "1")
-                self.assertIs(posixpath.islink(test_support.TESTFN + "2"), True)
-                self.assertIs(posixpath.exists(test_support.TESTFN + "2"), False)
-                self.assertIs(posixpath.lexists(test_support.TESTFN + "2"), True)
-        finally:
-            if not f.close():
-                f.close()
-            try:
-                os.remove(test_support.TESTFN + "1")
-            except os.error:
-                pass
-            try:
-                os.remove(test_support.TESTFN + "2")
-            except os.error:
-                pass
-
-        self.assertRaises(TypeError, posixpath.islink)
-
-    def test_exists(self):
-        self.assertIs(posixpath.exists(test_support.TESTFN), False)
-        f = open(test_support.TESTFN, "wb")
-        try:
-            f.write("foo")
-            f.close()
-            self.assertIs(posixpath.exists(test_support.TESTFN), True)
-            self.assertIs(posixpath.lexists(test_support.TESTFN), True)
-        finally:
-            if not f.close():
-                f.close()
-            try:
-                os.remove(test_support.TESTFN)
-            except os.error:
-                pass
-
-        self.assertRaises(TypeError, posixpath.exists)
-
-    def test_isdir(self):
-        self.assertIs(posixpath.isdir(test_support.TESTFN), False)
-        f = open(test_support.TESTFN, "wb")
-        try:
-            f.write("foo")
-            f.close()
-            self.assertIs(posixpath.isdir(test_support.TESTFN), False)
-            os.remove(test_support.TESTFN)
-            os.mkdir(test_support.TESTFN)
-            self.assertIs(posixpath.isdir(test_support.TESTFN), True)
-            os.rmdir(test_support.TESTFN)
-        finally:
-            if not f.close():
-                f.close()
-            try:
-                os.remove(test_support.TESTFN)
-            except os.error:
-                pass
-            try:
-                os.rmdir(test_support.TESTFN)
-            except os.error:
-                pass
-
-        self.assertRaises(TypeError, posixpath.isdir)
-
-    def test_isfile(self):
-        self.assertIs(posixpath.isfile(test_support.TESTFN), False)
-        f = open(test_support.TESTFN, "wb")
-        try:
-            f.write("foo")
-            f.close()
-            self.assertIs(posixpath.isfile(test_support.TESTFN), True)
-            os.remove(test_support.TESTFN)
-            os.mkdir(test_support.TESTFN)
-            self.assertIs(posixpath.isfile(test_support.TESTFN), False)
-            os.rmdir(test_support.TESTFN)
-        finally:
-            if not f.close():
-                f.close()
-            try:
-                os.remove(test_support.TESTFN)
-            except os.error:
-                pass
-            try:
-                os.rmdir(test_support.TESTFN)
-            except os.error:
-                pass
-
-        self.assertRaises(TypeError, posixpath.isdir)
-
-        def test_samefile(self):
-            f = open(test_support.TESTFN + "1", "wb")
-            try:
-                f.write("foo")
-                f.close()
-                self.assertIs(
-                    posixpath.samefile(
-                        test_support.TESTFN + "1",
-                        test_support.TESTFN + "1"
-                    ),
-                    True
-                )
-                # If we don't have links, assume that os.stat doesn't return resonable
-                # inode information and thus, that samefile() doesn't work
-                if hasattr(os, "symlink"):
-                    os.symlink(
-                        test_support.TESTFN + "1",
-                        test_support.TESTFN + "2"
-                    )
-                    self.assertIs(
-                        posixpath.samefile(
-                            test_support.TESTFN + "1",
-                            test_support.TESTFN + "2"
-                        ),
-                        True
-                    )
-                    os.remove(test_support.TESTFN + "2")
-                    f = open(test_support.TESTFN + "2", "wb")
-                    f.write("bar")
-                    f.close()
-                    self.assertIs(
-                        posixpath.samefile(
-                            test_support.TESTFN + "1",
-                            test_support.TESTFN + "2"
-                        ),
-                        False
-                    )
-            finally:
-                if not f.close():
-                    f.close()
-                try:
-                    os.remove(test_support.TESTFN + "1")
-                except os.error:
-                    pass
-                try:
-                    os.remove(test_support.TESTFN + "2")
-                except os.error:
-                    pass
-
-            self.assertRaises(TypeError, posixpath.samefile)
-
-    if os.name != 'java':
-        def test_samestat(self):
-            f = open(test_support.TESTFN + "1", "wb")
-            try:
-                f.write("foo")
-                f.close()
-                self.assertIs(
-                    posixpath.samestat(
-                        os.stat(test_support.TESTFN + "1"),
-                        os.stat(test_support.TESTFN + "1")
-                    ),
-                    True
-                )
-                # If we don't have links, assume that os.stat() doesn't return resonable
-                # inode information and thus, that samefile() doesn't work
-                if hasattr(os, "symlink"):
-                    if hasattr(os, "symlink"):
-                        os.symlink(test_support.TESTFN + "1", test_support.TESTFN + "2")
-                        self.assertIs(
-                            posixpath.samestat(
-                                os.stat(test_support.TESTFN + "1"),
-                                os.stat(test_support.TESTFN + "2")
-                            ),
-                            True
-                        )
-                        os.remove(test_support.TESTFN + "2")
-                    f = open(test_support.TESTFN + "2", "wb")
-                    f.write("bar")
-                    f.close()
-                    self.assertIs(
-                        posixpath.samestat(
-                            os.stat(test_support.TESTFN + "1"),
-                            os.stat(test_support.TESTFN + "2")
-                        ),
-                        False
-                    )
-            finally:
-                if not f.close():
-                    f.close()
-                try:
-                    os.remove(test_support.TESTFN + "1")
-                except os.error:
-                    pass
-                try:
-                    os.remove(test_support.TESTFN + "2")
-                except os.error:
-                    pass
-
-            self.assertRaises(TypeError, posixpath.samestat)
-
-        def test_ismount(self):
-            self.assertIs(posixpath.ismount("/"), True)
-
-            self.assertRaises(TypeError, posixpath.ismount)
-
-    def test_expanduser(self):
-        self.assertEqual(posixpath.expanduser("foo"), "foo")
-        try:
-            import pwd
-        except ImportError:
-            pass
-        else:
-            self.assert_(isinstance(posixpath.expanduser("~/"), basestring))
-            # if home directory == root directory, this test makes no sense
-            if posixpath.expanduser("~") != '/':
-                self.assertEqual(
-                    posixpath.expanduser("~") + "/",
-                    posixpath.expanduser("~/")
-                )
-            self.assert_(isinstance(posixpath.expanduser("~root/"), basestring))
-            self.assert_(isinstance(posixpath.expanduser("~foo/"), basestring))
-
-        self.assertRaises(TypeError, posixpath.expanduser)
-
-    def test_expandvars(self):
-        oldenv = os.environ.copy()
-        try:
-            os.environ.clear()
-            os.environ["foo"] = "bar"
-            os.environ["{foo"] = "baz1"
-            os.environ["{foo}"] = "baz2"
-            self.assertEqual(posixpath.expandvars("foo"), "foo")
-            self.assertEqual(posixpath.expandvars("$foo bar"), "bar bar")
-            self.assertEqual(posixpath.expandvars("${foo}bar"), "barbar")
-            self.assertEqual(posixpath.expandvars("$[foo]bar"), "$[foo]bar")
-            self.assertEqual(posixpath.expandvars("$bar bar"), "$bar bar")
-            self.assertEqual(posixpath.expandvars("$?bar"), "$?bar")
-            self.assertEqual(posixpath.expandvars("${foo}bar"), "barbar")
-            self.assertEqual(posixpath.expandvars("$foo}bar"), "bar}bar")
-            self.assertEqual(posixpath.expandvars("${foo"), "${foo")
-            self.assertEqual(posixpath.expandvars("${{foo}}"), "baz1}")
-        finally:
-            os.environ.clear()
-            os.environ.update(oldenv)
-
-        self.assertRaises(TypeError, posixpath.expandvars)
-
-    def test_normpath(self):
-        self.assertEqual(posixpath.normpath(""), ".")
-        self.assertEqual(posixpath.normpath("/"), "/")
-        self.assertEqual(posixpath.normpath("//"), "//")
-        self.assertEqual(posixpath.normpath("///"), "/")
-        self.assertEqual(posixpath.normpath("///foo/.//bar//"), "/foo/bar")
-        self.assertEqual(posixpath.normpath("///foo/.//bar//.//..//.//baz"), "/foo/baz")
-        self.assertEqual(posixpath.normpath("///..//./foo/.//bar"), "/foo/bar")
-
-        self.assertRaises(TypeError, posixpath.normpath)
-
-    def test_abspath(self):
-        self.assert_("foo" in posixpath.abspath("foo"))
-
-        self.assertRaises(TypeError, posixpath.abspath)
-
-    def test_realpath(self):
-        self.assert_("foo" in realpath("foo"))
-        self.assertRaises(TypeError, posixpath.realpath)
-
-    if hasattr(os, "symlink"):
-        @unittest.skip("FIXME: broken")
-        def test_realpath_basic(self):
-            # Basic operation.
-            try:
-                os.symlink(ABSTFN+"1", ABSTFN)
-                self.assertEqual(realpath(ABSTFN), ABSTFN+"1")
-            finally:
-                self.safe_remove(ABSTFN)
-
-        def test_realpath_symlink_loops(self):
-            # Bug #930024, return the path unchanged if we get into an infinite
-            # symlink loop.
-            try:
-                old_path = abspath('.')
-                os.symlink(ABSTFN, ABSTFN)
-                self.assertEqual(realpath(ABSTFN), ABSTFN)
-
-                os.symlink(ABSTFN+"1", ABSTFN+"2")
-                os.symlink(ABSTFN+"2", ABSTFN+"1")
-                self.assertEqual(realpath(ABSTFN+"1"), ABSTFN+"1")
-                self.assertEqual(realpath(ABSTFN+"2"), ABSTFN+"2")
-
-                # Test using relative path as well.
-                os.chdir(dirname(ABSTFN))
-                self.assertEqual(realpath(basename(ABSTFN)), ABSTFN)
-            finally:
-                os.chdir(old_path)
-                self.safe_remove(ABSTFN)
-                self.safe_remove(ABSTFN+"1")
-                self.safe_remove(ABSTFN+"2")
-
-        @unittest.skip("FIXME: broken")
-        def test_realpath_resolve_parents(self):
-            # We also need to resolve any symlinks in the parents of a relative
-            # path passed to realpath. E.g.: current working directory is
-            # /usr/doc with 'doc' being a symlink to /usr/share/doc. We call
-            # realpath("a"). This should return /usr/share/doc/a/.
-            try:
-                old_path = abspath('.')
-                os.mkdir(ABSTFN)
-                os.mkdir(ABSTFN + "/y")
-                os.symlink(ABSTFN + "/y", ABSTFN + "/k")
-
-                os.chdir(ABSTFN + "/k")
-                self.assertEqual(realpath("a"), ABSTFN + "/y/a")
-            finally:
-                os.chdir(old_path)
-                self.safe_remove(ABSTFN + "/k")
-                self.safe_rmdir(ABSTFN + "/y")
-                self.safe_rmdir(ABSTFN)
-
-        @unittest.skip("FIXME: broken")
-        def test_realpath_resolve_before_normalizing(self):
-            # Bug #990669: Symbolic links should be resolved before we
-            # normalize the path. E.g.: if we have directories 'a', 'k' and 'y'
-            # in the following hierarchy:
-            # a/k/y
-            #
-            # and a symbolic link 'link-y' pointing to 'y' in directory 'a',
-            # then realpath("link-y/..") should return 'k', not 'a'.
-            try:
-                old_path = abspath('.')
-                os.mkdir(ABSTFN)
-                os.mkdir(ABSTFN + "/k")
-                os.mkdir(ABSTFN + "/k/y")
-                os.symlink(ABSTFN + "/k/y", ABSTFN + "/link-y")
-
-                # Absolute path.
-                self.assertEqual(realpath(ABSTFN + "/link-y/.."), ABSTFN + "/k")
-                # Relative path.
-                os.chdir(dirname(ABSTFN))
-                self.assertEqual(realpath(basename(ABSTFN) + "/link-y/.."), ABSTFN + "/k")
-            finally:
-                os.chdir(old_path)
-                self.safe_remove(ABSTFN + "/link-y")
-                self.safe_rmdir(ABSTFN + "/k/y")
-                self.safe_rmdir(ABSTFN + "/k")
-                self.safe_rmdir(ABSTFN)
-
-        @unittest.skip("FIXME: broken")
-        def test_realpath_resolve_first(self):
-            # Bug #1213894: The first component of the path, if not absolute,
-            # must be resolved too.
-
-            try:
-                old_path = abspath('.')
-                os.mkdir(ABSTFN)
-                os.mkdir(ABSTFN + "/k")
-                os.symlink(ABSTFN, ABSTFN + "link")
-                os.chdir(dirname(ABSTFN))
-
-                base = basename(ABSTFN)
-                self.assertEqual(realpath(base + "link"), ABSTFN)
-                self.assertEqual(realpath(base + "link/k"), ABSTFN + "/k")
-            finally:
-                os.chdir(old_path)
-                self.safe_remove(ABSTFN + "link")
-                self.safe_rmdir(ABSTFN + "/k")
-                self.safe_rmdir(ABSTFN)
-
-        # Convenience functions for removing temporary files.
-        def pass_os_error(self, func, filename):
-            try: func(filename)
-            except OSError: pass
-
-        def safe_remove(self, filename):
-            self.pass_os_error(os.remove, filename)
-
-        def safe_rmdir(self, dirname):
-            self.pass_os_error(os.rmdir, dirname)
-
-def test_main():
-    test_support.run_unittest(PosixPathTest)
-
-if __name__=="__main__":
-    test_main()

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 14 04:08:19 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 14 Mar 2015 03:08:19 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Exceptions_with_non-ascii_a?=
 =?utf-8?q?rgs_should_not_cause_in_exception_when_displayed?=
Message-ID: <20150314030819.86369.87160@psf.io>

https://hg.python.org/jython/rev/e23b59d070c0
changeset:   7612:e23b59d070c0
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 13 21:08:02 2015 -0600
summary:
  Exceptions with non-ascii args should not cause in exception when displayed

sys.excepthook, via Py#displayException, was failing with
UnicodeEncodeError when printing such exceptions (since that's the
behavior of __str__). Now conforms to what CPython does in this case:
only display the exception type if any exception is raised by __str__.

files:
  Lib/test/test_exceptions_jy.py         |  22 +++++++++++++-
  src/org/python/core/Py.java            |   6 +++-
  src/org/python/core/PyException.java   |   4 +-
  src/org/python/core/StdoutWrapper.java |   4 +-
  4 files changed, 30 insertions(+), 6 deletions(-)


diff --git a/Lib/test/test_exceptions_jy.py b/Lib/test/test_exceptions_jy.py
--- a/Lib/test/test_exceptions_jy.py
+++ b/Lib/test/test_exceptions_jy.py
@@ -2,9 +2,10 @@
 
 Made for Jython.
 """
-from test import test_support
+import sys
 import unittest
 from javatests import StackOverflowErrorTest
+from test import test_support
 
 
 class C:
@@ -56,6 +57,25 @@
             cm.exception.message,
             "maximum recursion depth exceeded (Java StackOverflowError)")
 
+    def test_unicode_args(self):
+        e = RuntimeError(u"Drink \u2615")  # coffee emoji
+        # Can take the repr of any object
+        self.assertEqual(repr(e), "RuntimeError(u'Drink \u2615',)")
+        # Cannot of course turn a non-ascii Unicode object into a str, even if it's an exception object
+        with self.assertRaises(UnicodeEncodeError) as cm:
+            str(e)
+        self.assertEqual(
+            str(cm.exception),
+            "'ascii' codec can't encode character u'\u2615' in position 6: ordinal not in range(128)")
+        # But the exception hook, via Py#displayException, does not fail when attempting to __str__ the exception args
+        with test_support.captured_stderr() as s:
+            sys.excepthook(RuntimeError, u"Drink \u2615", None)
+        self.assertEqual(s.getvalue(), "RuntimeError\n")  
+        # It is fine with ascii values, of course
+        with test_support.captured_stderr() as s:
+            sys.excepthook(RuntimeError, u"Drink java", None)
+        self.assertEqual(s.getvalue(), "RuntimeError: Drink java\n")  
+
 
 def test_main():
     test_support.run_unittest(ExceptionsTestCase)
diff --git a/src/org/python/core/Py.java b/src/org/python/core/Py.java
--- a/src/org/python/core/Py.java
+++ b/src/org/python/core/Py.java
@@ -1185,7 +1185,11 @@
                 stderr.println(getStackTrace((Throwable) javaError));
             }
         }
-        stderr.println(formatException(type, value));
+        try {
+            stderr.println(formatException(type, value));
+        } catch (Exception ex) {
+            stderr.println(formatException(type, Py.None));
+        }
     }
 
     /**
diff --git a/src/org/python/core/PyException.java b/src/org/python/core/PyException.java
--- a/src/org/python/core/PyException.java
+++ b/src/org/python/core/PyException.java
@@ -3,7 +3,7 @@
 import java.io.*;
 
 /**
- * A wrapper for all python exception. Note that the wellknown python exception are <b>not</b>
+ * A wrapper for all python exception. Note that the well-known python exceptions are <b>not</b>
  * subclasses of PyException. Instead the python exception class is stored in the <code>type</code>
  * field and value or class instance is stored in the <code>value</code> field.
  */
@@ -58,7 +58,7 @@
     }
 
     public PyException(PyObject type, String value) {
-        this(type, new PyString(value));
+        this(type, Py.newStringOrUnicode(value));
     }
 
     private boolean printingStackTrace = false;
diff --git a/src/org/python/core/StdoutWrapper.java b/src/org/python/core/StdoutWrapper.java
--- a/src/org/python/core/StdoutWrapper.java
+++ b/src/org/python/core/StdoutWrapper.java
@@ -248,11 +248,11 @@
     }
 
     public void print(String s) {
-        print(new PyString(s), false, false);
+        print(Py.newStringOrUnicode(s), false, false);
     }
 
     public void println(String s) {
-        print(new PyString(s), false, true);
+        print(Py.newStringOrUnicode(s), false, true);
     }
 
     public void print(PyObject o) {

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 14 12:57:30 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 14 Mar 2015 11:57:30 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Comment_out_SRE=5FSTATE=23T?=
 =?utf-8?q?RACE_debug_method_in_regex_matching?=
Message-ID: <20150314115730.88852.21035@psf.io>

https://hg.python.org/jython/rev/28c98fa8978a
changeset:   7613:28c98fa8978a
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sat Mar 14 05:57:06 2015 -0600
summary:
  Comment out SRE_STATE#TRACE debug method in regex matching

Avoids unnecessary overhead and fixes http://bugs.jython.org/issue1725

files:
  src/org/python/modules/sre/SRE_STATE.java |  114 +++++-----
  1 files changed, 57 insertions(+), 57 deletions(-)


diff --git a/src/org/python/modules/sre/SRE_STATE.java b/src/org/python/modules/sre/SRE_STATE.java
--- a/src/org/python/modules/sre/SRE_STATE.java
+++ b/src/org/python/modules/sre/SRE_STATE.java
@@ -270,13 +270,13 @@
                 newsize = 512;
                 if (newsize < minsize)
                     newsize = minsize;
-                TRACE(0, ptr, "allocate stack " + newsize);
+//                TRACE(0, ptr, "allocate stack " + newsize);
                 stack = new int[newsize];
             } else {
                 /* grow the stack */
                 while (newsize < minsize)
                     newsize += newsize;
-                TRACE(0, ptr, "grow stack to " + newsize);
+//                TRACE(0, ptr, "grow stack to " + newsize);
                 stack = new int[newsize];
                 System.arraycopy(mark_stack, 0, stack, 0, mark_stack.length);
             }
@@ -284,7 +284,7 @@
             mark_stack_size = newsize;
         }
 
-        TRACE(0, ptr, "copy " + lo + ":" + hi + " to " + mark_stack_base + " (" + size + ")");
+//        TRACE(0, ptr, "copy " + lo + ":" + hi + " to " + mark_stack_base + " (" + size + ")");
 
         System.arraycopy(mark, lo, mark_stack, mark_stack_base, size);
 
@@ -301,7 +301,7 @@
 
         this.mark_stack_base = mark_stack_base - size;
 
-        TRACE(0, ptr, "copy " + lo + ":" + hi +  " from " + mark_stack_base);
+//        TRACE(0, ptr, "copy " + lo + ":" + hi +  " from " + mark_stack_base);
 
         System.arraycopy(mark_stack, this.mark_stack_base, mark, lo, size);
     }
@@ -374,11 +374,11 @@
             switch (set[setidx++]) {
 
             case SRE_OP_FAILURE:
-                TRACE(setidx, ch, "CHARSET FAILURE");
+//                TRACE(setidx, ch, "CHARSET FAILURE");
                 return !ok;
                 
             case SRE_OP_LITERAL:
-                TRACE(setidx, ch, "CHARSET LITERAL " + set[setidx]);
+//                TRACE(setidx, ch, "CHARSET LITERAL " + set[setidx]);
                 /* <LITERAL> <code> */
                 if (ch == set[setidx])
                     return ok;
@@ -387,14 +387,14 @@
                 
             case SRE_OP_CATEGORY:
                 /* <CATEGORY> <code> */
-                TRACE(setidx, ch, "CHARSET CHARSET " + set[setidx]);
+//                TRACE(setidx, ch, "CHARSET CHARSET " + set[setidx]);
                 if (sre_category(set[setidx], ch))
                     return ok;
                 setidx++;
                 break;
 
             case SRE_OP_CHARSET:
-                TRACE(setidx, ch, "CHARSET CHARSET ");
+//                TRACE(setidx, ch, "CHARSET CHARSET ");
                 /* <CHARSET> <bitmap> (16 bits per code word) */
 //                if (ch < 256 &&
 //                            (set[setidx + (ch >> 4)] & (1 << (ch & 15))) != 0)
@@ -409,20 +409,20 @@
                 
             case SRE_OP_RANGE:
                 /* <RANGE> <lower> <upper> */
-                TRACE(setidx, ch, "CHARSET RANGE " + set[setidx] + " " + set[setidx+1]);
+//                TRACE(setidx, ch, "CHARSET RANGE " + set[setidx] + " " + set[setidx+1]);
                 if (set[setidx] <= ch && ch <= set[setidx+1])
                     return ok;
                 setidx += 2;
                 break;
 
             case SRE_OP_NEGATE:
-                TRACE(setidx, ch, "CHARSET NEGATE");
+//                TRACE(setidx, ch, "CHARSET NEGATE");
                 ok = !ok;
                 break;
                 
             case SRE_OP_BIGCHARSET:
                 /* <BIGCHARSET> <blockcount> <256 blockindices> <blocks> */
-                TRACE(setidx, ch, "CHARSET BIGCHARSET ");
+//                TRACE(setidx, ch, "CHARSET BIGCHARSET ");
                 
 //                count = *(set++);
 //                if (!(ch & ~65535))
@@ -470,14 +470,14 @@
 
         case SRE_OP_IN:
             /* repeated set */
-            TRACE(pidx, ptr, "COUNT IN");
+//            TRACE(pidx, ptr, "COUNT IN");
             while (ptr < end && SRE_CHARSET(pattern, pidx + 2, str[ptr]))
                 ptr++;
             break;
             
         case SRE_OP_ANY:
             /* repeated dot wildcard. */
-            TRACE(pidx, ptr, "COUNT ANY");
+//            TRACE(pidx, ptr, "COUNT ANY");
             while (ptr < end && !SRE_IS_LINEBREAK(str[ptr]))
                 ptr++;
             break;
@@ -485,14 +485,14 @@
         case SRE_OP_ANY_ALL:
             /* repeated dot wildcard.  skip to the end of the target
                string, and backtrack from there */
-            TRACE(pidx, ptr, "COUNT ANY_ALL");
+//            TRACE(pidx, ptr, "COUNT ANY_ALL");
             ptr = end;
             break;
 
         case SRE_OP_LITERAL:
             /* repeated literal */
             chr = pattern[pidx+1];
-            TRACE(pidx, ptr, "COUNT LITERAL " + chr);
+//            TRACE(pidx, ptr, "COUNT LITERAL " + chr);
             while (ptr < end && str[ptr] == chr)
                 ptr++;
             break;
@@ -500,7 +500,7 @@
         case SRE_OP_LITERAL_IGNORE:
             /* repeated literal */
             chr = pattern[pidx+1];
-            TRACE(pidx, ptr, "COUNT LITERAL_IGNORE " + chr);
+//            TRACE(pidx, ptr, "COUNT LITERAL_IGNORE " + chr);
             while (ptr < end && lower(str[ptr]) == chr)
                 ptr++;
             break;
@@ -508,7 +508,7 @@
         case SRE_OP_NOT_LITERAL:
             /* repeated non-literal */
             chr = pattern[pidx+1];
-            TRACE(pidx, ptr, "COUNT NOT_LITERAL " + chr);
+//            TRACE(pidx, ptr, "COUNT NOT_LITERAL " + chr);
             while (ptr < end && str[ptr] != chr)
                 ptr++;
             break;
@@ -516,7 +516,7 @@
         case SRE_OP_NOT_LITERAL_IGNORE:
             /* repeated non-literal */
             chr = pattern[pidx+1];
-            TRACE(pidx, ptr, "COUNT NOT_LITERAL_IGNORE " + chr);
+//            TRACE(pidx, ptr, "COUNT NOT_LITERAL_IGNORE " + chr);
             while (ptr < end && lower(str[ptr]) != chr)
                 ptr++;
             break;
@@ -524,7 +524,7 @@
 
         default:
             /* repeated single character pattern */
-            TRACE(pidx, ptr, "COUNT SUBPATTERN");
+//            TRACE(pidx, ptr, "COUNT SUBPATTERN");
             while (this.ptr < end) {
                 i = SRE_MATCH(pattern, pidx, level);
                 if (i < 0)
@@ -548,7 +548,7 @@
 
         int lastmark, lastindex, mark_stack_base = 0;
 
-        TRACE(pidx, ptr, "ENTER " + level);
+//        TRACE(pidx, ptr, "ENTER " + level);
 
         if (level > USE_RECURSION_LIMIT)
            return SRE_ERROR_RECURSION_LIMIT;
@@ -570,7 +570,7 @@
             case SRE_OP_MARK:
                 /* set mark */
                 /* <MARK> <gid> */
-                TRACE(pidx, ptr, "MARK " + pattern[pidx]);
+//                TRACE(pidx, ptr, "MARK " + pattern[pidx]);
                 i = pattern[pidx];
                 if ((i & 1) != 0)
                     this.lastindex = i / 2 + 1;
@@ -583,7 +583,7 @@
             case SRE_OP_LITERAL:
                 /* match literal character */
                 /* <LITERAL> <code> */
-                TRACE(pidx, ptr, "LITERAL " + pattern[pidx]);
+//                TRACE(pidx, ptr, "LITERAL " + pattern[pidx]);
 
                 if (ptr >= end || str[ptr] != pattern[pidx])
                     return 0;
@@ -594,7 +594,7 @@
             case SRE_OP_NOT_LITERAL:
                 /* match anything that is not literal character */
                 /* args: <code> */
-                TRACE(pidx, ptr, "NOT_LITERAL " + pattern[pidx]);
+//                TRACE(pidx, ptr, "NOT_LITERAL " + pattern[pidx]);
                 if (ptr >= end || str[ptr] == pattern[pidx])
                     return 0;
                 pidx++;
@@ -603,14 +603,14 @@
                
             case SRE_OP_SUCCESS:
                 /* end of pattern */
-                TRACE(pidx, ptr, "SUCCESS");
+//                TRACE(pidx, ptr, "SUCCESS");
                 this.ptr = ptr;
                 return 1;
 
             case SRE_OP_AT:
                 /* match at given position */
                 /* <AT> <code> */
-                TRACE(pidx, ptr, "AT " + pattern[pidx]);
+//                TRACE(pidx, ptr, "AT " + pattern[pidx]);
                 if (!SRE_AT(ptr, pattern[pidx]))
                     return 0;
                 pidx++;
@@ -619,7 +619,7 @@
             case SRE_OP_CATEGORY:
                 /* match at given category */
                 /* <CATEGORY> <code> */
-                TRACE(pidx, ptr, "CATEGORY " + pattern[pidx]);
+//                TRACE(pidx, ptr, "CATEGORY " + pattern[pidx]);
 
                 if (ptr >= end || !sre_category(pattern[pidx], str[ptr]))
                     return 0;
@@ -630,7 +630,7 @@
 
             case SRE_OP_ANY:
                 /* match anything */
-                TRACE(pidx, ptr, "ANY");
+//                TRACE(pidx, ptr, "ANY");
                 if (ptr >= end || SRE_IS_LINEBREAK(str[ptr]))
                     return 0;
                 ptr++;
@@ -639,7 +639,7 @@
             case SRE_OP_ANY_ALL:
                 /* match anything */
                 /* <ANY_ALL> */
-                TRACE(pidx, ptr, "ANY_ALL");
+//                TRACE(pidx, ptr, "ANY_ALL");
                 if (ptr >= end)
                     return 0;
                 ptr++;
@@ -648,7 +648,7 @@
             case SRE_OP_IN:
                 /* match set member (or non_member) */
                 /* <IN> <skip> <set> */
-                TRACE(pidx, ptr, "IN");
+//                TRACE(pidx, ptr, "IN");
                 if (ptr >= end || !SRE_CHARSET(pattern, pidx + 1, str[ptr]))
                     return 0;
                 pidx += pattern[pidx];
@@ -656,7 +656,7 @@
                 break;
 
             case SRE_OP_LITERAL_IGNORE:
-                TRACE(pidx, ptr, "LITERAL_IGNORE " + pattern[pidx]);
+//                TRACE(pidx, ptr, "LITERAL_IGNORE " + pattern[pidx]);
                 if (ptr >= end || lower(str[ptr]) != lower(pattern[pidx]))
                     return 0;
                 pidx++;
@@ -664,7 +664,7 @@
                 break;
 
             case SRE_OP_NOT_LITERAL_IGNORE:
-                TRACE(pidx, ptr, "NOT_LITERAL_IGNORE " + pattern[pidx]);
+//                TRACE(pidx, ptr, "NOT_LITERAL_IGNORE " + pattern[pidx]);
                 if (ptr >= end || lower(str[ptr]) == lower(pattern[pidx]))
                     return 0;
                 pidx++;
@@ -672,7 +672,7 @@
                 break;
 
             case SRE_OP_IN_IGNORE:
-                TRACE(pidx, ptr, "IN_IGNORE");
+//                TRACE(pidx, ptr, "IN_IGNORE");
                 if (ptr >= end ||
                         !SRE_CHARSET(pattern, pidx + 1, lower(str[ptr])))
                     return 0;
@@ -685,7 +685,7 @@
             case SRE_OP_INFO:
                 /* jump forward */
                 /* <JUMP> <offset> */
-                TRACE(pidx, ptr, "JUMP " + pattern[pidx]);
+//                TRACE(pidx, ptr, "JUMP " + pattern[pidx]);
                 pidx += pattern[pidx];
                 break;
 
@@ -730,7 +730,7 @@
 
                 int mincount = pattern[pidx+1];
 
-                TRACE(pidx, ptr, "REPEAT_ONE " + mincount + " " + pattern[pidx+2]);
+//                TRACE(pidx, ptr, "REPEAT_ONE " + mincount + " " + pattern[pidx+2]);
                 if (ptr + mincount > end)
                     return 0; /* cannot match */
 
@@ -808,7 +808,7 @@
                 /* <MIN_REPEAT_ONE> <skip> <1=min> <2=max> item <SUCCESS> tail */
 
 
-                TRACE(pidx, ptr, "MIN_REPEAT_ONE");
+//                TRACE(pidx, ptr, "MIN_REPEAT_ONE");
 
                 if (ptr + pattern[pidx+1] > end)
                     return 0; /* cannot match */
@@ -866,7 +866,7 @@
                    by the UNTIL operator (MAX_UNTIL, MIN_UNTIL) */
                 /* <REPEAT> <skip> <1=min> <2=max> item <UNTIL> tail */
 
-                TRACE(pidx, ptr, "REPEAT " + pattern[pidx+1] + " " + pattern[pidx+2]);
+//                TRACE(pidx, ptr, "REPEAT " + pattern[pidx+1] + " " + pattern[pidx+2]);
 
                 SRE_REPEAT rep = new SRE_REPEAT(repeat);
                 rep.count = -1;
@@ -894,7 +894,7 @@
 
                 count = rp.count + 1;
 
-                TRACE(pidx, ptr, "MAX_UNTIL " + count);
+//                TRACE(pidx, ptr, "MAX_UNTIL " + count);
 
                 if (count < pattern[rp.pidx + 1]) {
                     /* not enough matches */
@@ -949,7 +949,7 @@
 
                 count = rp.count + 1;
 
-               TRACE(pidx, ptr, "MIN_UNTIL " + count + " " + rp.pidx);
+//               TRACE(pidx, ptr, "MIN_UNTIL " + count + " " + rp.pidx);
 
                 if (count < pattern[rp.pidx + 1]) {
                     /* not enough matches */
@@ -993,7 +993,7 @@
             case SRE_OP_GROUPREF:
                 /* match backreference */
                 i = pattern[pidx];
-                TRACE(pidx, ptr, "GROUPREF " + i);
+//                TRACE(pidx, ptr, "GROUPREF " + i);
                 int p = mark[i+i];
                 int e = mark[i+i+1];
                 if (p == -1 || e == -1 || e < p)
@@ -1010,7 +1010,7 @@
             case SRE_OP_GROUPREF_IGNORE:
                 /* match backreference */
                 i = pattern[pidx];
-                TRACE(pidx, ptr, "GROUPREF_IGNORE " + i);
+//                TRACE(pidx, ptr, "GROUPREF_IGNORE " + i);
                 p = mark[i+i];
                 e = mark[i+i+1];
                 if (p == -1 || e == -1 || e < p)
@@ -1026,7 +1026,7 @@
                 
             case SRE_OP_GROUPREF_EXISTS:
                 i = pattern[pidx];
-                TRACE(pidx, ptr, "GROUPREF_EXISTS " + i);
+//                TRACE(pidx, ptr, "GROUPREF_EXISTS " + i);
                 p = mark[i+i];
                 e = mark[i+i+1];
                 if (p == -1 || e == -1 || e < p) {
@@ -1039,7 +1039,7 @@
             case SRE_OP_ASSERT:
                 /* assert subpattern */
                 /* args: <skip> <back> <pattern> */
-                TRACE(pidx, ptr, "ASSERT " + pattern[pidx+1]);
+//                TRACE(pidx, ptr, "ASSERT " + pattern[pidx+1]);
 
                 this.ptr = ptr - pattern[pidx + 1];
                 if (this.ptr < this.beginning)
@@ -1053,7 +1053,7 @@
             case SRE_OP_ASSERT_NOT:
                 /* assert not subpattern */
                 /* args: <skip> <pattern> */
-                TRACE(pidx, ptr, "ASSERT_NOT " + pattern[pidx]);
+//                TRACE(pidx, ptr, "ASSERT_NOT " + pattern[pidx]);
                 this.ptr = ptr - pattern[pidx + 1];
                 if (this.ptr >= this.beginning) {
                     i = SRE_MATCH(pattern, pidx + 2, level + 1);
@@ -1067,11 +1067,11 @@
                 
             case SRE_OP_FAILURE:
                 /* immediate failure */
-                TRACE(pidx, ptr, "FAILURE");
+//                TRACE(pidx, ptr, "FAILURE");
                 return 0;
 
             default:
-                TRACE(pidx, ptr, "UNKNOWN " + pattern[pidx-1]);
+//                TRACE(pidx, ptr, "UNKNOWN " + pattern[pidx-1]);
                 return SRE_ERROR_ILLEGAL;
             }
         }
@@ -1145,7 +1145,7 @@
                     } else {
                         if (++i == prefix_len) {
                             /* found a potential match */
-                            TRACE(pidx, ptr, "SEARCH SCAN " + prefix_skip + " " + prefix_len);
+//                            TRACE(pidx, ptr, "SEARCH SCAN " + prefix_skip + " " + prefix_len);
                             this.start = ptr + 1 - prefix_len;
                             this.ptr = ptr + 1 - prefix_len + prefix_skip;
                             if ((flags & SRE_INFO_LITERAL) != 0)
@@ -1175,7 +1175,7 @@
                     ptr++;
                 if (ptr == end)
                     return 0;
-                TRACE(pidx, ptr, "SEARCH LITERAL");
+//                TRACE(pidx, ptr, "SEARCH LITERAL");
                 this.start = ptr;
                 this.ptr = ++ptr;
                 if ((flags & SRE_INFO_LITERAL) != 0)
@@ -1193,7 +1193,7 @@
                     ptr++;
                 if (ptr == end)
                     return 0;
-                TRACE(pidx, ptr, "SEARCH CHARSET");
+//                TRACE(pidx, ptr, "SEARCH CHARSET");
                 this.start = ptr;
                 this.ptr = ptr;
                 status = SRE_MATCH(pattern, pidx, 1);
@@ -1205,7 +1205,7 @@
         } else {
             /* general case */
             while (ptr <= end) {
-                TRACE(pidx, ptr, "SEARCH");
+//                TRACE(pidx, ptr, "SEARCH");
                 this.start = this.ptr = ptr++;
                 status = SRE_MATCH(pattern, pidx, 1);
                 if (status != 0)
@@ -1363,12 +1363,12 @@
         mark_fini();
     }
 
-    // XXX - this should get hot-spotted out, but for now useful for doing further optimization
-    // of this code (by comparing to CPython 2.5.2)
-    private static final boolean do_trace = false;
-    private void TRACE(int pidx, int ptr, String string) {
-        if (do_trace) {
-            System.out.println("      |" + pidx + "|" + Integer.toHexString(ptr) + ": " + string);
-        }
-    }
+    // XXX - this should get hot-spotted out, but that can take some time. Devs should just re-enable.
+    // See http://bugs.jython.org/issue1725
+//    private static final boolean do_trace = false;
+//    private void TRACE(int pidx, int ptr, String string) {
+//        if (do_trace) {
+//            System.out.println("      |" + pidx + "|" + Integer.toHexString(ptr) + ": " + string);
+//        }
+//    }
 }

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 14 14:44:03 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 14 Mar 2015 13:44:03 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Matches_CPython=27s_re_supp?=
	=?utf-8?q?ort_of_Unicode_whitespace?=
Message-ID: <20150314134403.88854.25040@psf.io>

https://hg.python.org/jython/rev/3ee1feff962d
changeset:   7614:3ee1feff962d
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sat Mar 14 07:27:26 2015 -0600
summary:
  Matches CPython's re support of Unicode whitespace

Fixes http://bugs.jython.org/issue2226

files:
  Lib/test/test_re_jy.py                    |  45 ++++++++++-
  src/org/python/modules/sre/SRE_STATE.java |   4 +-
  2 files changed, 45 insertions(+), 4 deletions(-)


diff --git a/Lib/test/test_re_jy.py b/Lib/test/test_re_jy.py
--- a/Lib/test/test_re_jy.py
+++ b/Lib/test/test_re_jy.py
@@ -1,6 +1,10 @@
 import re
+import sys
 import unittest
 import test.test_support
+import unicodedata
+from unicodedata import category
+
 
 class ReTest(unittest.TestCase):
 
@@ -28,8 +32,45 @@
 
     def test_unkown_groupname(self):
         self.assertRaises(IndexError,
-                          re.match("(?P<int>\d+)\.(\d*)", '3.14').group,
-                          "misspelled")
+                          re.match(r'(?P<int>\d+)\.(\d*)', '3.14').group,
+                          'misspelled')
+
+    def test_whitespace(self):
+        # Test for http://bugs.jython.org/issue2226 - verify against cpython
+        ws_re = re.compile(r'\s')
+        not_ws_re = re.compile(r'\S')
+        cpython_ascii_whitespace = set(' \t\n\r\f\v')
+        for i in xrange(256):
+            c = chr(i)
+            if c in cpython_ascii_whitespace:
+                self.assertRegexpMatches(c, ws_re)
+                self.assertNotRegexpMatches(c, not_ws_re)
+            else:
+                self.assertNotRegexpMatches(c, ws_re)
+                self.assertRegexpMatches(c, not_ws_re)
+
+    def test_unicode_whitespace(self):
+        # Test for http://bugs.jython.org/issue2226
+        ws_re = re.compile(r'\s', re.UNICODE)
+        not_ws_re = re.compile(r'\S', re.UNICODE)
+        separator_categories = set(['Zl', 'Zp', 'Zs'])
+        separators = {chr(c) for c in [28, 29, 30, 31]}
+        special = set([
+            unicodedata.lookup('MONGOLIAN VOWEL SEPARATOR'),
+            u'\u0085', # NEXT LINE (NEL)
+            ])
+        cpython_whitespace = set(' \t\n\r\f\v') | separators | special
+        for i in xrange(0xFFFF): # could test to sys.maxunicode, but does not appear to be necessary
+            if i >= 0xD800 and i <= 0xDFFF:
+                continue
+            c = unichr(i)
+            if c in cpython_whitespace or category(c) in separator_categories:
+                self.assertRegexpMatches(c, ws_re)
+                self.assertNotRegexpMatches(c, not_ws_re)
+            else:
+                self.assertNotRegexpMatches(c, ws_re)
+                self.assertRegexpMatches(c, not_ws_re)
+
 
 def test_main():
     test.test_support.run_unittest(ReTest)
diff --git a/src/org/python/modules/sre/SRE_STATE.java b/src/org/python/modules/sre/SRE_STATE.java
--- a/src/org/python/modules/sre/SRE_STATE.java
+++ b/src/org/python/modules/sre/SRE_STATE.java
@@ -230,9 +230,9 @@
             return !Character.isDigit(ch);
 
         case SRE_CATEGORY_UNI_SPACE:
-            return Character.isWhitespace(ch);
+            return Character.isSpaceChar(ch) || Character.isWhitespace(ch) || ch == 0x0085;
         case SRE_CATEGORY_UNI_NOT_SPACE:
-            return !Character.isWhitespace(ch);
+            return !(Character.isSpaceChar(ch) || Character.isWhitespace(ch) || ch == 0x0085);
 
         case SRE_CATEGORY_UNI_WORD:
             return Character.isLetterOrDigit(ch) || ch == '_';

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sun Mar 15 19:47:57 2015
From: jython-checkins at python.org (stefan.richthofer)
Date: Sun, 15 Mar 2015 18:47:57 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Added_guide_to_implement_Tr?=
 =?utf-8?q?averseproc_with_code_samples?=
Message-ID: <20150315184757.27379.82090@psf.io>

https://hg.python.org/jython/rev/d9ed7d9ba06a
changeset:   7615:d9ed7d9ba06a
user:        Stefan Richthofer <stefan.richthofer at gmx.de>
date:        Sun Mar 15 19:47:38 2015 +0100
summary:
  Added guide to implement Traverseproc with code samples
to doc of Traverseproc.java.

files:
  src/org/python/core/BaseSet.java       |   10 +-
  src/org/python/core/PySystemState.java |   21 -
  src/org/python/core/Traverseproc.java  |  239 +++++++++++-
  src/org/python/modules/gc.java         |    8 +-
  4 files changed, 226 insertions(+), 52 deletions(-)


diff --git a/src/org/python/core/BaseSet.java b/src/org/python/core/BaseSet.java
--- a/src/org/python/core/BaseSet.java
+++ b/src/org/python/core/BaseSet.java
@@ -606,9 +606,11 @@
     public int traverse(Visitproc visit, Object arg) {
         int retValue;
         for (PyObject ob: _set) {
-            retValue = visit.visit(ob, arg);
-            if (retValue != 0) {
-                return retValue;
+            if (ob != null) {
+                retValue = visit.visit(ob, arg);
+                if (retValue != 0) {
+                    return retValue;
+                }
             }
         }
         return 0;
@@ -616,6 +618,6 @@
 
     @Override
     public boolean refersDirectlyTo(PyObject ob) {
-        return _set.contains(ob);
+        return ob != null && _set.contains(ob);
     }
 }
diff --git a/src/org/python/core/PySystemState.java b/src/org/python/core/PySystemState.java
--- a/src/org/python/core/PySystemState.java
+++ b/src/org/python/core/PySystemState.java
@@ -1660,27 +1660,6 @@
     /* Traverseproc implementation */
     @Override
     public int traverse(Visitproc visit, Object arg) {
-//      Potential PyObject refs in PySystemState:
-//      public PyList argv = new PyList();
-//      public PyObject modules;
-//      public PyList path;
-//      public PyList warnoptions = new PyList();
-//      public PyObject builtins;
-//      public PyObject platform = defaultPlatform;
-//      public PyList meta_path;
-//      public PyList path_hooks;
-//      public PyObject path_importer_cache;
-//      public PyObject ps1 = new PyString(">>> ");
-//      public PyObject ps2 = new PyString("... ");
-//      public PyObject executable;
-//      public PyObject stdout, stderr, stdin;
-//      public PyObject __stdout__, __stderr__, __stdin__;
-//      public PyObject __displayhook__, __excepthook__;
-//      public PyObject last_value = Py.None;
-//      public PyObject last_type = Py.None;
-//      public PyObject last_traceback = Py.None;
-//      public PyObject __name__ = new PyString("sys");
-//      public PyObject __dict__;
         int retVal;
         if (argv != null) {
             retVal = visit.visit(argv, arg);
diff --git a/src/org/python/core/Traverseproc.java b/src/org/python/core/Traverseproc.java
--- a/src/org/python/core/Traverseproc.java
+++ b/src/org/python/core/Traverseproc.java
@@ -1,13 +1,16 @@
 package org.python.core;
 
+import org.python.expose.ExposedType;
+import org.python.modules.gc;
+
 /**
  * <p>
  * This interface defines a
  * <a href="https://docs.python.org/2.7/c-api/gcsupport.html" target="_blank">
- * CPython equivalent traverse mechanism
+ * CPython-equivalent traverse-mechanism
  * </a> allowing to detect reference cycles. While this is crucial for cyclic
- * gc support in CPython, it only serves debugging purposes in Jython. As a side
- * effect it allows a more complete implementation of the gc module.
+ * gc support in CPython, it only serves debugging purposes in Jython. As a
+ * side-effect it allows a more complete implementation of the gc module.
  * </p>
  * <p>
  * Note that implementing this interface is only OPTIONAL.<b> Gc will work fine
@@ -16,13 +19,13 @@
  * custom PyObject-implementations.
  * </p>
  * <p>
- * Of course this interface shall only be implemented by PyObjects that
- * potentially own direct references to other PyObjects. Note that indirect
+ * Of course this interface shall only be implemented by {@code PyObject}s that
+ * potentially own direct references to other {@code PyObject}s. Note that indirect
  * references via non-PyObjects should also be treated as "direct" (c.f.
  * tracefunc in {@link org.python.core.PyFrame}).
- * PyObjects that don't own references to other PyObjects under any condition
- * and neither inherit such references from a superclass are strictly recommended
- * to be annotated {@link org.python.core.Untraversable}.
+ * {@code PyObject}s that don't own references to other {@code PyObject}s under any
+ * condition and neither inherit such references from a superclass are strictly
+ * recommended to be annotated {@link org.python.core.Untraversable}.
  * </p>
  * <p>
  * Jython's traverse mechanism serves debugging purposes to ease finding memory
@@ -31,19 +34,19 @@
  * from these different behaviors. Jython's traverse mechanism is intended to
  * allow finding such bugs by comparing gc behavior to CPython and isolating
  * the Python code that is not robust enough to work invariant under different
- * gc behaviors.
+ * gc behaviors. See also {@link org.python.modules.gc} for more details on this.
  * </p>
  * <p>
  * Further this mechanism is crucial for some aspects of gc-support of the
  * <a href="http://www.jyni.org" target="_blank">JyNI</a>
  * project. JyNI does not strictly depend on it to emulate CPython's gc
  * for extensions, but would have to perform inefficient reflection-based
- * traversal in some edge-cases (which might also conflict security managers).
+ * traversal in some edge-cases (which might also conflict with security managers).
  * </p>
  * <p>
- * Note that the slots-array and - if existent - the user-dict of fooDerived classes
- * is traversed by {@link org.python.core.TraverseProcDerived}.
- * The gc module takes care of exploiting both traverse methods in its static traverse
+ * Note that the slots-array and - if existent - the user-dict of {@code fooDerived}
+ * classes is traversed by {@link org.python.core.TraverseProcDerived}.
+ * The gc-module takes care of exploiting both traverse methods in its static traverse
  * method. So for manual traversion one should always use
  * {@link org.python.modules.gc#traverse(PyObject, Visitproc, Object)} rather
  * than directly calling methods in this interface.
@@ -57,25 +60,209 @@
  * traversed (along with the user dict).
  * </p>
  * <p>
- * Note for implementing:<br>
- * Every non-static strong referenced PyObject should be passed to the
+ * <b>Note for implementing:</b><br>
+ * Every non-static, strong-referenced {@code PyObject} should be passed to the
  * {@link org.python.core.Visitproc}. If {@code Object}s or {@code interface}-types are
  * referenced where it is not known, whether it is a {@code PyObject} or
  * references other {@code PyObjects}, one should check for {@code PyObject}
- * via {@code instanceof} and otherwise also check whether it at least
- * implements {@code Traverseproc} itself. In latter case one should traverse
- * it by delegating to its {@code Traverseproc} methods.<br>
- * Warning:<br>
+ * via {@code instanceof}. If a non-{@code PyObject}
+ * implements {@code Traverseproc}, one can traverse
+ * it by delegating to its {@code Traverseproc} methods.<br><br>
+ * <b>Warning:</b><br>
  * If one lets non-{@code PyObject}s implement {@code Traverseproc}, extreme
  * care must be taken, whether the traverse call shall be passed on to other
  * non-{@code PyObject} {@code Traverseproc}-implementers, as this can cause
  * infinite traverse cycles.<br>
  * Examples for non-{@code PyObject}s that implement {@code Traverseproc} are
  * {@link org.python.core.PyException} and {@link com.ziclix.python.sql.Fetch}.
+ * A safer, but potentially slower way to deal with
+ * non-{@code PyObject}-{@code Traverseproc}s or any other non-{@code PyObject}
+ * that might contain references to other {@code PyObject}s is
+ * {@link org.python.modules.gc#traverseByReflection(Object, Visitproc, Object)}.
+ * This is for instance used in {@link org.python.core.PyArray}.
  * </p>
  * <p>
- * It follows a list of PyObject subclasses in Jython, excluding derived classes.<br>
- * PyObject subclasses in Jython checked for need of Traverseproc:<br>
+ * <br>
+ * <b>Examples</b><br><br>
+ * In the following we provide some examples with code-snippets to demonstrate
+ * and streamline the writing of {@code Traverseproc}-implementations.<br>
+ * Since this peace of API was introduced to enhance a large existing
+ * code-base, we recommend to put the {@code Traverseproc}-implementation always
+ * to the end of a class and separate it from the original code by two blank
+ * lines and a comment "Traverseproc implementation".<br><br>
+ * Let's start with classes that don't hold references to {@code PyObject}s.
+ * If the class extends some other class that implements {@code Traverseproc}, nothing
+ * special needs to be done. For instance, we have this situation in
+ * {@link org.python.core.PySet}. It extends {@link org.python.core.BaseSet},
+ * which in turn implements {@code Traverseproc}:<br><br>
+ * <pre>
+ * {@literal @}ExposedType(name = "set", base = PyObject.class, doc = BuiltinDocs.set_doc)
+ * public class PySet extends BaseSet {
+ *   ...
+ * }
+ * </pre>
+ * If the class neither contains {@code PyObject}-references, nor extends some
+ * {@code Traverseproc}-implementing class, it is recommended to be annotated
+ * {@link org.python.core.Untraversable}. {@link org.python.core.PyInteger} is
+ * an example for this:<br><br>
+ * <pre>
+ * {@literal @}Untraversable
+ * {@literal @}ExposedType(name = "int", doc = BuiltinDocs.int_doc)
+ * public class PyInteger extends PyObject {
+ *   ...
+ * }
+ * </pre>
+ * If there are simply some {@code PyObject}(-subclass), non-static fields in the class,
+ * let it implement {@link org.python.core.Traverseproc}.
+ * Write {@link org.python.core.Traverseproc#traverse(Visitproc, Object)} by
+ * just visiting the fields one by one. Check each to be non-{@code null} previously
+ * unless the field cannot be {@code null} for some good reason. If
+ * {@link org.python.core.Visitproc#visit(PyObject, Object)} returns non-zero,
+ * return the result immediately (i.e. abort the traverse process).
+ * The following example is taken from
+ * {@link org.python.core.PyMethod}:<br><br>
+ * <pre>
+ *  /{@literal *} Traverseproc implementation {@literal *}/
+ *  {@literal @}Override
+ *  public int traverse(Visitproc visit, Object arg) {
+ *      int retVal;
+ *      if (im_class != null) {
+ *          retVal = visit.visit(im_class, arg);
+ *          if (retVal != 0) {
+ *              return retVal;
+ *          }
+ *      }
+ *      if (__func__ != null) {
+ *          retVal = visit.visit(__func__, arg);
+ *          if (retVal != 0) {
+ *              return retVal;
+ *          }
+ *      }
+ *      return __self__ == null ? 0 : visit.visit(__self__, arg);
+ *  }
+ * </pre>
+ * Implement {@link org.python.core.Traverseproc#refersDirectlyTo(PyObject)}
+ * by checking the argument to be non-{@code null} and identity-comparing it to
+ * every field:<br><br>
+ * <pre>
+ *  {@literal @}Override
+ *  public boolean refersDirectlyTo(PyObject ob) {
+ *      return ob != null && (ob == im_class || ob == __func__ || ob == __self__);
+ *  }
+ * </pre>
+ * If there is a Java-set or other iterable that it is not a {@code PyObject},
+ * but contains {@code PyObject}s, visit every element. Don't forget to check
+ * for non-{@code null} if necessary and return immediately, if
+ * {@link org.python.core.Visitproc#visit(PyObject, Object)} returns non-zero.
+ * The following example is taken from {@link org.python.core.BaseSet}:<br><br>
+ * <pre>
+ *  /{@literal *} Traverseproc implementation {@literal *}/
+ *  {@literal @}Override
+ *  public int traverse(Visitproc visit, Object arg) {
+ *      int retValue;
+ *      for (PyObject ob: _set) {
+ *          if (ob != null) {
+ *              retValue = visit.visit(ob, arg);
+ *              if (retValue != 0) {
+ *                  return retValue;
+ *              }
+ *          }
+ *      }
+ *      return 0;
+ *  }
+ * </pre>
+ * In this case, {@link org.python.core.Traverseproc#refersDirectlyTo(PyObject)}
+ * can be implemented (potentially) efficiently by using the backing set's
+ * {@code contains}-method:<br><br>
+ * <pre>
+ *  {@literal @}Override
+ *  public boolean refersDirectlyTo(PyObject ob) {
+ *      return ob != null && _set.contains(ob);
+ *  }
+ * </pre>
+ * If a class extends a {@code Traverseproc}-implementing class and adds
+ * {@code PyObject}-references to it, the parent-{@code traverse}-method
+ * should be called initially via {@code super} (example is taken from
+ * {@link org.python.core.PyJavaType}):<br><br>
+ * <pre>
+ *  /{@literal *} Traverseproc implementation {@literal *}/
+ *  {@literal @}Override
+ *  public int traverse(Visitproc visit, Object arg) {
+ *      int retVal = super.traverse(visit, arg);
+ *      if (retVal != 0) {
+ *          return retVal;
+ *      }
+ *      if (conflicted != null) {
+ *          for (PyObject ob: conflicted) {
+ *              if (ob != null) {
+ *                  retVal = visit.visit(ob, arg);
+ *                  if (retVal != 0) {
+ *                      return retVal;
+ *                  }
+ *              }
+ *          }
+ *      }
+ *      return 0;
+ *  }
+ * </pre>
+ * In contrast to that, {@link org.python.core.Traverseproc#refersDirectlyTo(PyObject)}
+ * should call its parent-method as late as possible, because that method might throw an
+ * {@code UnsupportedOperationException}. By calling it in the end, we have the chance
+ * to fail- or succeed fast before a potential exception occurs:<br><br>
+ * <pre>
+ *  {@literal @}Override
+ *  public boolean refersDirectlyTo(PyObject ob) throws UnsupportedOperationException {
+ *      if (ob == null) {
+ *          return false;
+ *      }
+ *      if (conflicted != null) {
+ *          for (PyObject obj: conflicted) {
+ *              if (obj == ob) {
+ *                  return true;
+ *              }
+ *          }
+ *      }
+ *      return super.refersDirectlyTo(ob);
+ *  }
+ * </pre>
+ * While reflection-based traversal should be avoided if possible, it can be used to
+ * traverse fields that might contain references to {@code PyObject}s, but cannot be
+ * inferred at compile-time.
+ * {@link org.python.modules.gc#canLinkToPyObject(Class, boolean)} can help to safe
+ * some performance by failing fast if type-info already rules out the possibility
+ * of the field holding {@code PyObject}-references.
+ * This technique is for instance used to traverse the content of
+ * {@link org.python.core.PyArray}:<br><br>
+ * <pre>
+ *  /{@literal *} Traverseproc implementation {@literal *}/
+ *  {@literal @}Override
+ *  public int traverse(Visitproc visit, Object arg) {
+ *      if (data == null || !gc.canLinkToPyObject(data.getClass(), true)) {
+ *          return 0;
+ *      }
+ *      return gc.traverseByReflection(data, visit, arg);
+ *  }
+ * </pre>
+ * {@link org.python.modules.gc#canLinkToPyObject(Class, boolean)} also
+ * offers a way to let {@link org.python.core.Traverseproc#refersDirectlyTo(PyObject)}
+ * fail fast by type-information:<br><br>
+ * <pre>
+ *  {@literal @}Override
+ *  public boolean refersDirectlyTo(PyObject ob)
+ *          throws UnsupportedOperationException {
+ *      if (data == null || !gc.canLinkToPyObject(data.getClass(), true)) {
+ *          return false;
+ *      }
+ *      throw new UnsupportedOperationException();
+ *  }
+ * </pre>
+ * </p>
+ * <p>
+ * <br>
+ * <b>List of {@code PyObject}-subclasses</b><br><br>
+ * We conclude with a list of {@code PyObject} subclasses in Jython, excluding
+ * derived classes.<br>
+ * {@code PyObject}-subclasses in Jython checked for need of {@code Traverseproc}:<br>
  * <br>
  * <br>
  * org.python.core:<br>
@@ -96,7 +283,6 @@
  *     BinFunction                 - no refs, untraversable<br>
  *   AstList                       - Traverseproc<br>
  *   BaseBytes                     - no refs, untraversable<br>
- *     IndexDelegate               - no PyObject<br>
  *   BaseDictionaryView            - Traverseproc<br>
  *   BaseSet                       - Traverseproc<br>
  *   ClasspathPyImporter           - no refs, untraversable<br>
@@ -269,7 +455,9 @@
  *   PyPartial                     - Traverseproc<br>
  * <br>
  * org.python.modules._io:<br>
- *   PyFileIO                      - no refs, untraversable (there is a final PyString "mode", which is guarenteed to be a PyString and no subclass; as such it needs not be traversed since it cannot have refs itself)<br>
+ *   PyFileIO                      - no refs, untraversable (there is a final PyString
+ *   "mode" that is guaranteed to be a PyString and no subclass; as such it needs not be
+ *   traversed since it cannot have refs itself)<br>
  *   PyIOBase                      - Traverseproc<br>
  *   PyRawIOBase                   - no refs, extends PyIOBase<br>
  * <br>
@@ -358,6 +546,9 @@
  *   Time:<br>
  *     TimeFunctions               - no refs, untraversable<br>
  * <br>
+ * org.python.modules.zipimport:<br>
+ *   zipimporter                   - Traverseproc<br>
+ * <br>
  * org.python.util:<br>
  *   InteractiveInterpreter        - no PyObject<br>
  * <br>
@@ -446,6 +637,8 @@
  * @see org.python.core.Untraversable
  * @see org.python.core.Visitproc
  * @see org.python.modules.gc#traverse(PyObject, Visitproc, Object)
+ * @see org.python.modules.gc#traverseByReflection(Object, Visitproc, Object)
+ * @see org.python.modules.gc#canLinkToPyObject(Class, boolean)
  */
 public interface Traverseproc {
 
diff --git a/src/org/python/modules/gc.java b/src/org/python/modules/gc.java
--- a/src/org/python/modules/gc.java
+++ b/src/org/python/modules/gc.java
@@ -27,7 +27,7 @@
 import org.python.modules._weakref.GlobalRef;
 
 //These imports belong to the out-commented section on MXBean-based
-//GC-sync far below. That section is kept to document this failed
+//gc-sync far below. That section is kept to document this failed
 //approach and allow easy reproduction of this failure.
 //import java.lang.management.*;
 //import javax.management.*;
@@ -97,8 +97,8 @@
  * Waiting for trash could in theory be strictly synchronized by using {@code MXBean}s, i.e.
  * <a href="https://docs.oracle.com/javase/7/docs/jre/api/management/extension/index.html?com/sun/management/GcInfo.html"
  * target="_blank">GarbageCollectionNotificationInfo</a> and related API.
- * However, experiments showed that the arising GC-notifications do not reliably indicate
- * when enqueuing was done for a specific GC-run. We kept the experimental implementation
+ * However, experiments showed that the arising gc-notifications do not reliably indicate
+ * when enqueuing was done for a specific gc-run. We kept the experimental implementation
  * in source-code comments to allow easy reproducibility of this issue. (Note that out-commented
  * code contradicts Jython-styleguide, but this one - however - is needed to document this
  * infeasible approach and is explicitly declared accordingly).<br>
@@ -2935,7 +2935,7 @@
      * This method checks via type-checking-only, whether an object
      * of the given class can in principle hold a ref to a {@code PyObject}.
      * Especially if arrays are involved, this can safe a lot performance.
-     * For now, no generic-type info is exploited.
+     * For now, no generic type-info is exploited.
      * </p>
      * <p>
      * If {@code actual} is true, the answer will hold for an object

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Mon Mar 16 05:39:22 2015
From: jython-checkins at python.org (jim.baker)
Date: Mon, 16 Mar 2015 04:39:22 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Use_Java_instead_of_JNR_for?=
 =?utf-8?q?_os=2Echmod=2C_os=2Emkdir_when_running_on_Windows?=
Message-ID: <20150316043922.86367.70829@psf.io>

https://hg.python.org/jython/rev/9b911ee72842
changeset:   7616:9b911ee72842
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sun Mar 15 22:39:18 2015 -0600
summary:
  Use Java instead of JNR for os.chmod, os.mkdir when running on Windows

By doing so, Jython gets UNC path support for the functions, which is
apparently lacking in the Windows C runtime library.  Also catches
SecurityException and reraises as OSError(Errno.EACCES).

Future fix may want to revisit if these and other os functions can be
readily rewritten to use Java NIO2 where possible, but this will
require further analysis of error conditions.

Fixes http://bugs.jython.org/issue2120

files:
  src/org/python/modules/posix/PosixModule.java |  57 ++++++---
  1 files changed, 39 insertions(+), 18 deletions(-)


diff --git a/src/org/python/modules/posix/PosixModule.java b/src/org/python/modules/posix/PosixModule.java
--- a/src/org/python/modules/posix/PosixModule.java
+++ b/src/org/python/modules/posix/PosixModule.java
@@ -87,7 +87,7 @@
     private static final int W_OK = 1 << 1;
     private static final int R_OK = 1 << 2;
 
-    /** Lazily initialzed singleton source for urandom. */
+    /** Lazily initialized singleton source for urandom. */
     private static class UrandomSource {
         static final SecureRandom INSTANCE = new SecureRandom();
     }
@@ -262,17 +262,11 @@
         if ((mode & W_OK) != 0 && !file.canWrite()) {
             result = false;
         }
-        if ((mode & X_OK) != 0) {
-            // NOTE: always true without native jna-posix stat
-            try {
-                result = posix.stat(absolutePath(path).toString()).isExecutable();
-            } catch (PyException pye) {
-                if (!pye.match(Py.OSError)) {
-                    throw pye;
-                }
-                // ENOENT
-                result = false;
-            }
+        if ((mode & X_OK) != 0 && !file.canExecute()) {
+            // Previously Jython used JNR Posix, but this is unnecessary -
+            // File#canExecute uses the same code path
+            // http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6379654
+            result = false;
         }
         return result;
     }
@@ -297,7 +291,16 @@
         "chmod(path, mode)\n\n" +
         "Change the access permissions of a file.");
     public static void chmod(PyObject path, int mode) {
-        if (posix.chmod(absolutePath(path).toString(), mode) < 0) {
+        if (os == OS.NT) {
+            try {
+                if (!absolutePath(path).toFile().setWritable((mode & FileStat.S_IWUSR) == 0)) {
+                    throw Py.OSError(Errno.EPERM, path);
+                }
+            } catch (SecurityException ex) {
+                throw Py.OSError(Errno.EACCES, path);
+            }
+
+        } else if (posix.chmod(absolutePath(path).toString(), mode) < 0) {
             throw errorFromErrno(path);
         }
     }
@@ -329,7 +332,7 @@
         int fd_high = getFD(fd_highObj).getIntFD(false);
         for (int i = fd_low; i < fd_high; i++) {
             try {
-                posix.close(i); // FIXME catch exceptions
+                posix.close(i);
             } catch (Exception e) {}
         }
     }
@@ -509,6 +512,7 @@
     public static PyString __doc__getpid = new PyString(
         "getpid() -> pid\n\n" +
         "Return the current process id");
+
     @Hide(posixImpl = PosixImpl.JAVA)
     public static int getpid() {
         return posix.getpid();
@@ -593,6 +597,7 @@
     public static PyString __doc__link = new PyString(
         "link(src, dst)\n\n" +
         "Create a hard link to a file.");
+
     @Hide(OS.NT)
     public static void link(PyObject src, PyObject dst) {
         if (posix.link(absolutePath(src).toString(), absolutePath(dst).toString()) < 0) {
@@ -651,7 +656,15 @@
     }
 
     public static void mkdir(PyObject path, int mode) {
-        if (posix.mkdir(absolutePath(path).toString(), mode) < 0) {
+        if (os == OS.NT) {
+            try {
+                if (!absolutePath(path).toFile().mkdir()) {
+                    throw Py.OSError(Errno.EPERM, path);
+                }
+            } catch (SecurityException ex) {
+                throw Py.OSError(Errno.EACCES, path);
+            }
+        } else if (posix.mkdir(absolutePath(path).toString(), mode) < 0) {
             throw errorFromErrno(path);
         }
     }
@@ -840,6 +853,7 @@
     public static PyString __doc__symlink = new PyString(
         "symlink(src, dst)\n\n" +
         "Create a symbolic link pointing to src named dst.");
+
     @Hide(OS.NT)
     public static void symlink(PyObject src, PyObject dst) {
         if (posix.symlink(asPath(src), absolutePath(dst).toString()) < 0) {
@@ -885,10 +899,10 @@
                 throw Py.OSError(Errno.EISDIR, path);
             } else if (!Files.deleteIfExists(nioPath)) {
                 // Something went wrong, does stat raise an error?
-                posix.stat(nioPath.toString());
+                basicstat(path, nioPath);
                 // It exists, do we not have permissions?
                 if (!Files.isWritable(nioPath)) {
-                    throw Py.OSError(Errno.EPERM, path);
+                    throw Py.OSError(Errno.EACCES, path);
                 }
                 throw Py.OSError("unlink(): an unknown error occurred: " + nioPath.toString());
             }
@@ -1137,6 +1151,8 @@
             throw Py.OSError(Errno.ENOENT, path);
         } catch (IOException ioe) {
             throw Py.OSError(Errno.EBADF, path);
+        } catch (SecurityException ex) {
+            throw Py.OSError(Errno.EACCES, path);
         }
     }
 
@@ -1170,6 +1186,8 @@
                 throw Py.OSError(Errno.ENOENT, path);
             } catch (IOException ioe) {
                 throw Py.OSError(Errno.EBADF, path);
+            } catch (SecurityException ex) {
+                throw Py.OSError(Errno.EACCES, path);
             }
         }
     }
@@ -1195,6 +1213,8 @@
                 throw Py.OSError(Errno.ENOENT, path);
             } catch (IOException ioe) {
                 throw Py.OSError(Errno.EBADF, path);
+            } catch (SecurityException ex) {
+                throw Py.OSError(Errno.EACCES, path);
             }
         }
     }
@@ -1253,6 +1273,8 @@
                 throw Py.OSError(Errno.ENOENT, path);
             } catch (IOException ioe) {
                 throw Py.OSError(Errno.EBADF, path);
+            } catch (SecurityException ex) {
+                throw Py.OSError(Errno.EACCES, path);
             }
         }
     }
@@ -1273,7 +1295,6 @@
                     stat = posix.fstat(fd.intFD);
                 } else {
                     stat = posix.fstat(fd.javaFD);
-                    ;
                 }
                 return PyStatResult.fromFileStat(stat);
             } catch (PyException ex) {

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Mon Mar 16 17:00:52 2015
From: jython-checkins at python.org (jim.baker)
Date: Mon, 16 Mar 2015 16:00:52 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Fix_minor_but_build-breakin?=
 =?utf-8?q?g_errors_on_Windows_from_7616=3A9b911ee72842?=
Message-ID: <20150316160052.88866.4137@psf.io>

https://hg.python.org/jython/rev/19eee6ac6d0a
changeset:   7617:19eee6ac6d0a
user:        Jim Baker <jim.baker at rackspace.com>
date:        Mon Mar 16 10:00:47 2015 -0600
summary:
  Fix minor but build-breaking errors on Windows from 7616:9b911ee72842

Better exception handling and correct parsing of S_IWUSR in chmod.
Completes fix for http://bugs.jython.org/issue2120

files:
  src/org/python/modules/posix/PosixModule.java |  16 +++++++--
  1 files changed, 12 insertions(+), 4 deletions(-)


diff --git a/src/org/python/modules/posix/PosixModule.java b/src/org/python/modules/posix/PosixModule.java
--- a/src/org/python/modules/posix/PosixModule.java
+++ b/src/org/python/modules/posix/PosixModule.java
@@ -12,6 +12,7 @@
 import java.nio.channels.Channel;
 import java.nio.channels.ClosedChannelException;
 import java.nio.channels.FileChannel;
+import java.nio.file.FileAlreadyExistsException;
 import java.nio.file.FileSystems;
 import java.nio.file.Files;
 import java.nio.file.LinkOption;
@@ -293,7 +294,7 @@
     public static void chmod(PyObject path, int mode) {
         if (os == OS.NT) {
             try {
-                if (!absolutePath(path).toFile().setWritable((mode & FileStat.S_IWUSR) == 0)) {
+                if (!absolutePath(path).toFile().setWritable((mode & FileStat.S_IWUSR) != 0)) {
                     throw Py.OSError(Errno.EPERM, path);
                 }
             } catch (SecurityException ex) {
@@ -658,9 +659,16 @@
     public static void mkdir(PyObject path, int mode) {
         if (os == OS.NT) {
             try {
-                if (!absolutePath(path).toFile().mkdir()) {
-                    throw Py.OSError(Errno.EPERM, path);
-                }
+                Path nioPath = absolutePath(path);
+                // Windows does not use any mode attributes in creating a directory;
+                // see the corresponding function in posixmodule.c, posix_mkdir;
+                // further work on mapping mode to PosixAttributes would have to be done
+                // for non Windows; posix.mkdir would still be necessary for mode bits like stat.S_ISGID
+                Files.createDirectory(nioPath);
+            } catch (FileAlreadyExistsException ex) {
+                throw Py.OSError(Errno.EEXIST, path);
+            } catch (IOException ioe) {
+                throw Py.OSError(ioe);
             } catch (SecurityException ex) {
                 throw Py.OSError(Errno.EACCES, path);
             }

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Tue Mar 17 21:57:48 2015
From: jython-checkins at python.org (jim.baker)
Date: Tue, 17 Mar 2015 20:57:48 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Posix_link_and_symlink_supp?=
	=?utf-8?q?ort_now_uses_NIO2?=
Message-ID: <20150317205747.63578.18319@psf.io>

https://hg.python.org/jython/rev/dedc704b8678
changeset:   7618:dedc704b8678
user:        Jim Baker <jim.baker at rackspace.com>
date:        Tue Mar 17 14:57:43 2015 -0600
summary:
  Posix link and symlink support now uses NIO2

Uses standard catch and translate to OSError with errno constants.
Fixes http://bugs.jython.org/issue2288

files:
  Lib/test/test_os_jy.py                        |  78 +++++++++-
  src/org/python/modules/posix/PosixModule.java |  40 ++++-
  2 files changed, 109 insertions(+), 9 deletions(-)


diff --git a/Lib/test/test_os_jy.py b/Lib/test/test_os_jy.py
--- a/Lib/test/test_os_jy.py
+++ b/Lib/test/test_os_jy.py
@@ -5,6 +5,7 @@
 Made for Jython.
 """
 import array
+import errno
 import glob
 import os
 import subprocess
@@ -293,6 +294,79 @@
                 "42\n47\n")
 
 
+ at unittest.skipUnless(hasattr(os, 'link'), "os.link not available")
+class LinkTestCase(unittest.TestCase):
+
+    def test_bad_link(self):
+        with test_support.temp_cwd() as new_cwd:
+            target = os.path.join(new_cwd, "target")
+            with open(target, "w") as f:
+                f.write("TARGET")
+            source = os.path.join(new_cwd, "source")
+            with self.assertRaises(OSError) as cm:
+                os.link(target, target)
+            self.assertEqual(cm.exception.errno, errno.EEXIST)
+
+            with self.assertRaises(OSError) as cm:
+                os.link("nonexistent-file", source)
+            self.assertEqual(cm.exception.errno, errno.ENOENT)
+
+    def test_link(self):
+        with test_support.temp_cwd() as new_cwd:
+            target = os.path.join(new_cwd, "target")
+            with open(target, "w") as f:
+                f.write("TARGET")
+            source = os.path.join(new_cwd, "source")
+            os.link(target, source)
+            with open(source, "r") as f:
+                self.assertEqual(f.read(), "TARGET")
+
+
+ at unittest.skipUnless(hasattr(os, 'symlink'), "symbolic link support  not available")
+class SymbolicLinkTestCase(unittest.TestCase):
+
+    def test_bad_symlink(self):
+        with test_support.temp_cwd() as new_cwd:
+            target = os.path.join(new_cwd, "target")
+            with open(target, "w") as f:
+                f.write("TARGET")
+            source = os.path.join(new_cwd, "source")
+            with self.assertRaises(OSError) as cm:
+                os.symlink(source, target)  # reversed args!
+            self.assertEqual(cm.exception.errno, errno.EEXIST)
+
+    def test_readlink(self):
+        with test_support.temp_cwd() as new_cwd:
+            target = os.path.join(new_cwd, "target")
+            with open(target, "w") as f:
+                f.write("TARGET")
+            source = os.path.join(new_cwd, "source")
+            os.symlink(target, source)
+            self.assertEqual(os.readlink(source), target)
+            self.assertEqual(os.readlink(unicode(source)), unicode(target))
+            self.assertIsInstance(os.readlink(unicode(source)), unicode)
+            
+    def test_readlink_non_symlink(self):
+        """os.readlink of a non symbolic link should raise an error"""
+        # test for http://bugs.jython.org/issue2292
+        with test_support.temp_cwd() as new_cwd:
+            target = os.path.join(new_cwd, "target")
+            with open(target, "w") as f:
+                f.write("TARGET")
+            with self.assertRaises(OSError) as cm:
+                os.readlink(target)
+            self.assertEqual(cm.exception.errno, errno.EINVAL)
+            self.assertEqual(cm.exception.filename, target)
+
+    def test_readlink_nonexistent(self):
+        with test_support.temp_cwd() as new_cwd:
+            nonexistent_file = os.path.join(new_cwd, "nonexistent-file")
+            with self.assertRaises(OSError) as cm:
+                os.readlink(nonexistent_file)
+            self.assertEqual(cm.exception.errno, errno.ENOENT)
+            self.assertEqual(cm.exception.filename, nonexistent_file)
+
+
 def test_main():
     test_support.run_unittest(
         OSFileTestCase, 
@@ -301,7 +375,9 @@
         OSWriteTestCase,
         UnicodeTestCase,
         LocaleTestCase,
-        SystemTestCase
+        SystemTestCase,
+        LinkTestCase,
+        SymbolicLinkTestCase,
     )
 
 if __name__ == '__main__':
diff --git a/src/org/python/modules/posix/PosixModule.java b/src/org/python/modules/posix/PosixModule.java
--- a/src/org/python/modules/posix/PosixModule.java
+++ b/src/org/python/modules/posix/PosixModule.java
@@ -16,8 +16,10 @@
 import java.nio.file.FileSystems;
 import java.nio.file.Files;
 import java.nio.file.LinkOption;
+import java.nio.file.NotLinkException;
 import java.nio.file.NoSuchFileException;
 import java.nio.file.Path;
+import java.nio.file.Paths;
 import java.nio.file.attribute.BasicFileAttributes;
 import java.nio.file.attribute.DosFileAttributes;
 import java.security.SecureRandom;
@@ -601,8 +603,17 @@
 
     @Hide(OS.NT)
     public static void link(PyObject src, PyObject dst) {
-        if (posix.link(absolutePath(src).toString(), absolutePath(dst).toString()) < 0) {
-            throw errorFromErrno();
+        try {
+            Files.createLink(Paths.get(asPath(dst)), Paths.get(asPath(src)));
+        } catch (FileAlreadyExistsException ex) {
+            throw Py.OSError(Errno.EEXIST);
+        } catch (NoSuchFileException ex) {
+            throw Py.OSError(Errno.ENOENT);
+        } catch (IOException ioe) {
+            System.err.println("Got this exception " + ioe);
+            throw Py.OSError(ioe);
+        } catch (SecurityException ex) {
+            throw Py.OSError(Errno.EACCES);
         }
     }
 
@@ -662,8 +673,6 @@
                 Path nioPath = absolutePath(path);
                 // Windows does not use any mode attributes in creating a directory;
                 // see the corresponding function in posixmodule.c, posix_mkdir;
-                // further work on mapping mode to PosixAttributes would have to be done
-                // for non Windows; posix.mkdir would still be necessary for mode bits like stat.S_ISGID
                 Files.createDirectory(nioPath);
             } catch (FileAlreadyExistsException ex) {
                 throw Py.OSError(Errno.EEXIST, path);
@@ -672,6 +681,9 @@
             } catch (SecurityException ex) {
                 throw Py.OSError(Errno.EACCES, path);
             }
+        // Further work on mapping mode to PosixAttributes would have to be done
+        // for non Windows platforms. In addition, posix.mkdir would still be necessary
+        // for mode bits like stat.S_ISGID
         } else if (posix.mkdir(absolutePath(path).toString(), mode) < 0) {
             throw errorFromErrno(path);
         }
@@ -780,11 +792,17 @@
         "readlink(path) -> path\n\n" +
         "Return a string representing the path to which the symbolic link points.");
     @Hide(OS.NT)
-    public static String readlink(PyObject path) {
+    public static PyString readlink(PyObject path) {
         try {
-            return posix.readlink(absolutePath(path).toString());
+            return Py.newStringOrUnicode(path, Files.readSymbolicLink(absolutePath(path)).toString());
+        } catch (NotLinkException ex) {
+            throw Py.OSError(Errno.EINVAL, path);
+        } catch (NoSuchFileException ex) {
+            throw Py.OSError(Errno.ENOENT, path);
         } catch (IOException ioe) {
             throw Py.OSError(ioe);
+        } catch (SecurityException ex) {
+            throw Py.OSError(Errno.EACCES, path);
         }
     }
 
@@ -864,8 +882,14 @@
 
     @Hide(OS.NT)
     public static void symlink(PyObject src, PyObject dst) {
-        if (posix.symlink(asPath(src), absolutePath(dst).toString()) < 0) {
-            throw errorFromErrno();
+        try {
+            Files.createSymbolicLink(Paths.get(asPath(dst)), Paths.get(asPath(src)));
+        } catch (FileAlreadyExistsException ex) {
+            throw Py.OSError(Errno.EEXIST);
+        } catch (IOException ioe) {
+            throw Py.OSError(ioe);
+        } catch (SecurityException ex) {
+            throw Py.OSError(Errno.EACCES);
         }
     }
 

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Thu Mar 19 18:14:47 2015
From: jython-checkins at python.org (jim.baker)
Date: Thu, 19 Mar 2015 17:14:47 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Re-applied_patch_for_file_e?=
 =?utf-8?q?xtension_of_compiled_Python_classes?=
Message-ID: <20150319171447.30042.47810@psf.io>

https://hg.python.org/jython/rev/57f8839eaac6
changeset:   7620:57f8839eaac6
user:        Jim Baker <jim.baker at rackspace.com>
date:        Thu Mar 19 11:14:34 2015 -0600
summary:
  Re-applied patch for file extension of compiled Python classes

The usual .pyc -> $py.class change necessary for Jython

files:
  Lib/zipfile.py |  4 +++-
  1 files changed, 3 insertions(+), 1 deletions(-)


diff --git a/Lib/zipfile.py b/Lib/zipfile.py
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -7,6 +7,8 @@
 import re
 import string
 
+_is_jython = os.name == "java"
+
 try:
     import zlib # We may need its compression method
     crc32 = zlib.crc32
@@ -1423,7 +1425,7 @@
         /python/lib/string, return (/python/lib/string.pyc, string).
         """
         file_py  = pathname + ".py"
-        file_pyc = pathname + ".pyc"
+        file_pyc = pathname + (".pyc" if not _is_jython else "$py.class")
         file_pyo = pathname + ".pyo"
         if os.path.isfile(file_pyo) and \
                             os.stat(file_pyo).st_mtime >= os.stat(file_py).st_mtime:

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Thu Mar 19 18:14:47 2015
From: jython-checkins at python.org (jim.baker)
Date: Thu, 19 Mar 2015 17:14:47 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Updated_zipfile_to_lib-pyth?=
	=?utf-8?q?on/2=2E7_version?=
Message-ID: <20150319171447.16938.65033@psf.io>

https://hg.python.org/jython/rev/a3bacacbb8cd
changeset:   7619:a3bacacbb8cd
user:        Jim Baker <jim.baker at rackspace.com>
date:        Thu Mar 19 11:13:41 2015 -0600
summary:
  Updated zipfile to lib-python/2.7 version

files:
  Lib/zipfile.py |  575 +++++++++++++++++++++---------------
  1 files changed, 328 insertions(+), 247 deletions(-)


diff --git a/Lib/zipfile.py b/Lib/zipfile.py
--- a/Lib/zipfile.py
+++ b/Lib/zipfile.py
@@ -5,6 +5,7 @@
 import binascii, cStringIO, stat
 import io
 import re
+import string
 
 try:
     import zlib # We may need its compression method
@@ -16,8 +17,6 @@
 __all__ = ["BadZipfile", "error", "ZIP_STORED", "ZIP_DEFLATED", "is_zipfile",
            "ZipInfo", "ZipFile", "PyZipFile", "LargeZipFile" ]
 
-is_jython = sys.platform.startswith('java')
-
 class BadZipfile(Exception):
     pass
 
@@ -168,6 +167,8 @@
         return endrec
 
     data = fpin.read(sizeEndCentDir64Locator)
+    if len(data) != sizeEndCentDir64Locator:
+        return endrec
     sig, diskno, reloff, disks = struct.unpack(structEndArchive64Locator, data)
     if sig != stringEndArchive64Locator:
         return endrec
@@ -178,6 +179,8 @@
     # Assume no 'zip64 extensible data'
     fpin.seek(offset - sizeEndCentDir64Locator - sizeEndCentDir64, 2)
     data = fpin.read(sizeEndCentDir64)
+    if len(data) != sizeEndCentDir64:
+        return endrec
     sig, sz, create_version, read_version, disk_num, disk_dir, \
             dircount, dircount2, dirsize, diroffset = \
             struct.unpack(structEndArchive64, data)
@@ -213,7 +216,9 @@
     except IOError:
         return None
     data = fpin.read()
-    if data[0:4] == stringEndArchive and data[-2:] == "\000\000":
+    if (len(data) == sizeEndCentDir and
+        data[0:4] == stringEndArchive and
+        data[-2:] == b"\000\000"):
         # the signature is correct and there's no comment, unpack structure
         endrec = struct.unpack(structEndArchive, data)
         endrec=list(endrec)
@@ -237,20 +242,21 @@
     if start >= 0:
         # found the magic number; attempt to unpack and interpret
         recData = data[start:start+sizeEndCentDir]
+        if len(recData) != sizeEndCentDir:
+            # Zip file is corrupted.
+            return None
         endrec = list(struct.unpack(structEndArchive, recData))
-        comment = data[start+sizeEndCentDir:]
-        # check that comment length is correct
-        if endrec[_ECD_COMMENT_SIZE] == len(comment):
-            # Append the archive comment and start offset
-            endrec.append(comment)
-            endrec.append(maxCommentStart + start)
+        commentSize = endrec[_ECD_COMMENT_SIZE] #as claimed by the zip file
+        comment = data[start+sizeEndCentDir:start+sizeEndCentDir+commentSize]
+        endrec.append(comment)
+        endrec.append(maxCommentStart + start)
 
-            # Try to read the "Zip64 end of central directory" structure
-            return _EndRecData64(fpin, maxCommentStart + start - filesize,
-                                 endrec)
+        # Try to read the "Zip64 end of central directory" structure
+        return _EndRecData64(fpin, maxCommentStart + start - filesize,
+                             endrec)
 
     # Unable to find a valid end of central directory structure
-    return
+    return None
 
 
 class ZipInfo (object):
@@ -294,6 +300,10 @@
 
         self.filename = filename        # Normalized file name
         self.date_time = date_time      # year, month, day, hour, min, sec
+
+        if date_time[0] < 1980:
+            raise ValueError('ZIP does not support timestamps before 1980')
+
         # Standard values:
         self.compress_type = ZIP_STORED # Type of compression for the file
         self.comment = ""               # Comment for each file
@@ -316,7 +326,7 @@
         # compress_size         Size of the compressed file
         # file_size             Size of the uncompressed file
 
-    def FileHeader(self):
+    def FileHeader(self, zip64=None):
         """Return the per-file header as a string."""
         dt = self.date_time
         dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2]
@@ -331,12 +341,17 @@
 
         extra = self.extra
 
-        if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT:
-            # File is larger than what fits into a 4 byte integer,
-            # fall back to the ZIP64 extension
+        if zip64 is None:
+            zip64 = file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT
+        if zip64:
             fmt = '<HHQQ'
             extra = extra + struct.pack(fmt,
                     1, struct.calcsize(fmt)-4, file_size, compress_size)
+        if file_size > ZIP64_LIMIT or compress_size > ZIP64_LIMIT:
+            if not zip64:
+                raise LargeZipFile("Filesize would require ZIP64 extensions")
+            # File is larger than what fits into a 4 byte integer,
+            # fall back to the ZIP64 extension
             file_size = 0xffffffff
             compress_size = 0xffffffff
             self.extract_version = max(45, self.extract_version)
@@ -461,6 +476,28 @@
         self._UpdateKeys(c)
         return c
 
+
+compressor_names = {
+    0: 'store',
+    1: 'shrink',
+    2: 'reduce',
+    3: 'reduce',
+    4: 'reduce',
+    5: 'reduce',
+    6: 'implode',
+    7: 'tokenize',
+    8: 'deflate',
+    9: 'deflate64',
+    10: 'implode',
+    12: 'bzip2',
+    14: 'lzma',
+    18: 'terse',
+    19: 'lz77',
+    97: 'wavpack',
+    98: 'ppmd',
+}
+
+
 class ZipExtFile(io.BufferedIOBase):
     """File-like object for reading an archive member.
        Is returned by ZipFile.open().
@@ -475,9 +512,11 @@
     # Search for universal newlines or line chunks.
     PATTERN = re.compile(r'^(?P<chunk>[^\r\n]+)|(?P<newline>\n|\r\n?)')
 
-    def __init__(self, fileobj, mode, zipinfo, decrypter=None):
+    def __init__(self, fileobj, mode, zipinfo, decrypter=None,
+            close_fileobj=False):
         self._fileobj = fileobj
         self._decrypter = decrypter
+        self._close_fileobj = close_fileobj
 
         self._compress_type = zipinfo.compress_type
         self._compress_size = zipinfo.compress_size
@@ -485,6 +524,12 @@
 
         if self._compress_type == ZIP_DEFLATED:
             self._decompressor = zlib.decompressobj(-15)
+        elif self._compress_type != ZIP_STORED:
+            descr = compressor_names.get(self._compress_type)
+            if descr:
+                raise NotImplementedError("compression type %d (%s)" % (self._compress_type, descr))
+            else:
+                raise NotImplementedError("compression type %d" % (self._compress_type,))
         self._unconsumed = ''
 
         self._readbuffer = ''
@@ -649,9 +694,15 @@
         self._offset += len(data)
         return data
 
+    def close(self):
+        try :
+            if self._close_fileobj:
+                self._fileobj.close()
+        finally:
+            super(ZipExtFile, self).close()
 
 
-class ZipFile:
+class ZipFile(object):
     """ Class with methods to open, read, write, close, list zip files.
 
     z = ZipFile(file, mode="r", compression=ZIP_STORED, allowZip64=False)
@@ -690,7 +741,7 @@
         self.compression = compression  # Method of compression
         self.mode = key = mode.replace('b', '')[0]
         self.pwd = None
-        self.comment = ''
+        self._comment = ''
 
         # Check if we were passed a file-like object
         if isinstance(file, basestring):
@@ -710,30 +761,34 @@
             self.fp = file
             self.filename = getattr(file, 'name', None)
 
-        if key == 'r':
-            self._GetContents()
-        elif key == 'w':
-            # set the modified flag so central directory gets written
-            # even if no files are added to the archive
-            self._didModify = True
-        elif key == 'a':
-            try:
-                # See if file is a zip file
+        try:
+            if key == 'r':
                 self._RealGetContents()
-                # seek to start of directory and overwrite
-                self.fp.seek(self.start_dir, 0)
-            except BadZipfile:
-                # file is not a zip file, just append
-                self.fp.seek(0, 2)
-
+            elif key == 'w':
                 # set the modified flag so central directory gets written
                 # even if no files are added to the archive
                 self._didModify = True
-        else:
+            elif key == 'a':
+                try:
+                    # See if file is a zip file
+                    self._RealGetContents()
+                    # seek to start of directory and overwrite
+                    self.fp.seek(self.start_dir, 0)
+                except BadZipfile:
+                    # file is not a zip file, just append
+                    self.fp.seek(0, 2)
+
+                    # set the modified flag so central directory gets written
+                    # even if no files are added to the archive
+                    self._didModify = True
+            else:
+                raise RuntimeError('Mode must be "r", "w" or "a"')
+        except:
+            fp = self.fp
+            self.fp = None
             if not self._filePassed:
-                self.fp.close()
-                self.fp = None
-            raise RuntimeError, 'Mode must be "r", "w" or "a"'
+                fp.close()
+            raise
 
     def __enter__(self):
         return self
@@ -741,17 +796,6 @@
     def __exit__(self, type, value, traceback):
         self.close()
 
-    def _GetContents(self):
-        """Read the directory, making sure we close the file if the format
-        is bad."""
-        try:
-            self._RealGetContents()
-        except BadZipfile:
-            if not self._filePassed:
-                self.fp.close()
-                self.fp = None
-            raise
-
     def _RealGetContents(self):
         """Read in the table of contents for the ZIP file."""
         fp = self.fp
@@ -765,7 +809,7 @@
             print endrec
         size_cd = endrec[_ECD_SIZE]             # bytes in central directory
         offset_cd = endrec[_ECD_OFFSET]         # offset of central directory
-        self.comment = endrec[_ECD_COMMENT]     # archive comment
+        self._comment = endrec[_ECD_COMMENT]    # archive comment
 
         # "concat" is zero, unless zip was concatenated to another file
         concat = endrec[_ECD_LOCATION] - size_cd - offset_cd
@@ -784,9 +828,11 @@
         total = 0
         while total < size_cd:
             centdir = fp.read(sizeCentralDir)
-            if centdir[0:4] != stringCentralDir:
-                raise BadZipfile, "Bad magic number for central directory"
+            if len(centdir) != sizeCentralDir:
+                raise BadZipfile("Truncated central directory")
             centdir = struct.unpack(structCentralDir, centdir)
+            if centdir[_CD_SIGNATURE] != stringCentralDir:
+                raise BadZipfile("Bad magic number for central directory")
             if self.debug > 2:
                 print centdir
             filename = fp.read(centdir[_CD_FILENAME_LENGTH])
@@ -845,9 +891,9 @@
             try:
                 # Read by chunks, to avoid an OverflowError or a
                 # MemoryError with very large embedded files.
-                f = self.open(zinfo.filename, "r")
-                while f.read(chunk_size):     # Check CRC-32
-                    pass
+                with self.open(zinfo.filename, "r") as f:
+                    while f.read(chunk_size):     # Check CRC-32
+                        pass
             except BadZipfile:
                 return zinfo.filename
 
@@ -864,6 +910,22 @@
         """Set default password for encrypted files."""
         self.pwd = pwd
 
+    @property
+    def comment(self):
+        """The comment text associated with the ZIP file."""
+        return self._comment
+
+    @comment.setter
+    def comment(self, comment):
+        # check for valid comment length
+        if len(comment) >= ZIP_MAX_COMMENT:
+            if self.debug:
+                print('Archive comment is too long; truncating to %d bytes'
+                        % ZIP_MAX_COMMENT)
+            comment = comment[:ZIP_MAX_COMMENT]
+        self._comment = comment
+        self._didModify = True
+
     def read(self, name, pwd=None):
         """Return file bytes (as a string) for name."""
         return self.open(name, "r", pwd).read()
@@ -880,62 +942,72 @@
         # given a file object in the constructor
         if self._filePassed:
             zef_file = self.fp
+            should_close = False
         else:
             zef_file = open(self.filename, 'rb')
+            should_close = True
 
-        # Make sure we have an info object
-        if isinstance(name, ZipInfo):
-            # 'name' is already an info object
-            zinfo = name
-        else:
-            # Get info object for name
-            zinfo = self.getinfo(name)
+        try:
+            # Make sure we have an info object
+            if isinstance(name, ZipInfo):
+                # 'name' is already an info object
+                zinfo = name
+            else:
+                # Get info object for name
+                zinfo = self.getinfo(name)
 
-        zef_file.seek(zinfo.header_offset, 0)
+            zef_file.seek(zinfo.header_offset, 0)
 
-        # Skip the file header:
-        fheader = zef_file.read(sizeFileHeader)
-        if fheader[0:4] != stringFileHeader:
-            raise BadZipfile, "Bad magic number for file header"
+            # Skip the file header:
+            fheader = zef_file.read(sizeFileHeader)
+            if len(fheader) != sizeFileHeader:
+                raise BadZipfile("Truncated file header")
+            fheader = struct.unpack(structFileHeader, fheader)
+            if fheader[_FH_SIGNATURE] != stringFileHeader:
+                raise BadZipfile("Bad magic number for file header")
 
-        fheader = struct.unpack(structFileHeader, fheader)
-        fname = zef_file.read(fheader[_FH_FILENAME_LENGTH])
-        if fheader[_FH_EXTRA_FIELD_LENGTH]:
-            zef_file.read(fheader[_FH_EXTRA_FIELD_LENGTH])
+            fname = zef_file.read(fheader[_FH_FILENAME_LENGTH])
+            if fheader[_FH_EXTRA_FIELD_LENGTH]:
+                zef_file.read(fheader[_FH_EXTRA_FIELD_LENGTH])
 
-        if fname != zinfo.orig_filename:
-            raise BadZipfile, \
-                      'File name in directory "%s" and header "%s" differ.' % (
-                          zinfo.orig_filename, fname)
+            if fname != zinfo.orig_filename:
+                raise BadZipfile, \
+                        'File name in directory "%s" and header "%s" differ.' % (
+                            zinfo.orig_filename, fname)
 
-        # check for encrypted flag & handle password
-        is_encrypted = zinfo.flag_bits & 0x1
-        zd = None
-        if is_encrypted:
-            if not pwd:
-                pwd = self.pwd
-            if not pwd:
-                raise RuntimeError, "File %s is encrypted, " \
-                      "password required for extraction" % name
+            # check for encrypted flag & handle password
+            is_encrypted = zinfo.flag_bits & 0x1
+            zd = None
+            if is_encrypted:
+                if not pwd:
+                    pwd = self.pwd
+                if not pwd:
+                    raise RuntimeError, "File %s is encrypted, " \
+                        "password required for extraction" % name
 
-            zd = _ZipDecrypter(pwd)
-            # The first 12 bytes in the cypher stream is an encryption header
-            #  used to strengthen the algorithm. The first 11 bytes are
-            #  completely random, while the 12th contains the MSB of the CRC,
-            #  or the MSB of the file time depending on the header type
-            #  and is used to check the correctness of the password.
-            bytes = zef_file.read(12)
-            h = map(zd, bytes[0:12])
-            if zinfo.flag_bits & 0x8:
-                # compare against the file type from extended local headers
-                check_byte = (zinfo._raw_time >> 8) & 0xff
-            else:
-                # compare against the CRC otherwise
-                check_byte = (zinfo.CRC >> 24) & 0xff
-            if ord(h[11]) != check_byte:
-                raise RuntimeError("Bad password for file", name)
+                zd = _ZipDecrypter(pwd)
+                # The first 12 bytes in the cypher stream is an encryption header
+                #  used to strengthen the algorithm. The first 11 bytes are
+                #  completely random, while the 12th contains the MSB of the CRC,
+                #  or the MSB of the file time depending on the header type
+                #  and is used to check the correctness of the password.
+                bytes = zef_file.read(12)
+                h = map(zd, bytes[0:12])
+                if zinfo.flag_bits & 0x8:
+                    # compare against the file type from extended local headers
+                    check_byte = (zinfo._raw_time >> 8) & 0xff
+                else:
+                    # compare against the CRC otherwise
+                    check_byte = (zinfo.CRC >> 24) & 0xff
+                if ord(h[11]) != check_byte:
+                    raise RuntimeError("Bad password for file", name)
 
-        return  ZipExtFile(zef_file, mode, zinfo, zd)
+            return ZipExtFile(zef_file, mode, zinfo, zd,
+                    close_fileobj=should_close)
+        except:
+            if should_close:
+                zef_file.close()
+            raise
 
     def extract(self, member, path=None, pwd=None):
         """Extract a member from the archive to the current working directory,
@@ -969,17 +1041,25 @@
         """
         # build the destination pathname, replacing
         # forward slashes to platform specific separators.
-        # Strip trailing path separator, unless it represents the root.
-        if (targetpath[-1:] in (os.path.sep, os.path.altsep)
-            and len(os.path.splitdrive(targetpath)[1]) > 1):
-            targetpath = targetpath[:-1]
+        arcname = member.filename.replace('/', os.path.sep)
 
-        # don't include leading "/" from file name if present
-        if member.filename[0] == '/':
-            targetpath = os.path.join(targetpath, member.filename[1:])
-        else:
-            targetpath = os.path.join(targetpath, member.filename)
+        if os.path.altsep:
+            arcname = arcname.replace(os.path.altsep, os.path.sep)
+        # interpret absolute pathname as relative, remove drive letter or
+        # UNC path, redundant separators, "." and ".." components.
+        arcname = os.path.splitdrive(arcname)[1]
+        arcname = os.path.sep.join(x for x in arcname.split(os.path.sep)
+                    if x not in ('', os.path.curdir, os.path.pardir))
+        if os.path.sep == '\\':
+            # filter illegal characters on Windows
+            illegal = ':<>|"?*'
+            table = string.maketrans(illegal, '_' * len(illegal))
+            arcname = arcname.translate(table)
+            # remove trailing dots
+            arcname = (x.rstrip('.') for x in arcname.split(os.path.sep))
+            arcname = os.path.sep.join(x for x in arcname if x)
 
+        targetpath = os.path.join(targetpath, arcname)
         targetpath = os.path.normpath(targetpath)
 
         # Create all upper directories if necessary.
@@ -992,11 +1072,9 @@
                 os.mkdir(targetpath)
             return targetpath
 
-        source = self.open(member, pwd=pwd)
-        target = file(targetpath, "wb")
-        shutil.copyfileobj(source, target)
-        source.close()
-        target.close()
+        with self.open(member, pwd=pwd) as source, \
+             file(targetpath, "wb") as target:
+            shutil.copyfileobj(source, target)
 
         return targetpath
 
@@ -1062,20 +1140,23 @@
             zinfo.CRC = 0
             self.filelist.append(zinfo)
             self.NameToInfo[zinfo.filename] = zinfo
-            self.fp.write(zinfo.FileHeader())
+            self.fp.write(zinfo.FileHeader(False))
             return
 
         with open(filename, "rb") as fp:
             # Must overwrite CRC and sizes with correct data later
             zinfo.CRC = CRC = 0
             zinfo.compress_size = compress_size = 0
-            zinfo.file_size = file_size = 0
-            self.fp.write(zinfo.FileHeader())
+            # Compressed size can be larger than uncompressed size
+            zip64 = self._allowZip64 and \
+                    zinfo.file_size * 1.05 > ZIP64_LIMIT
+            self.fp.write(zinfo.FileHeader(zip64))
             if zinfo.compress_type == ZIP_DEFLATED:
                 cmpr = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION,
                      zlib.DEFLATED, -15)
             else:
                 cmpr = None
+            file_size = 0
             while 1:
                 buf = fp.read(1024 * 8)
                 if not buf:
@@ -1095,11 +1176,16 @@
             zinfo.compress_size = file_size
         zinfo.CRC = CRC
         zinfo.file_size = file_size
-        # Seek backwards and write CRC and file sizes
+        if not zip64 and self._allowZip64:
+            if file_size > ZIP64_LIMIT:
+                raise RuntimeError('File size has increased during compressing')
+            if compress_size > ZIP64_LIMIT:
+                raise RuntimeError('Compressed size larger than uncompressed size')
+        # Seek backwards and write file header (which will now include
+        # correct CRC and file sizes)
         position = self.fp.tell()       # Preserve current position in file
-        self.fp.seek(zinfo.header_offset + 14, 0)
-        self.fp.write(struct.pack("<LLL", zinfo.CRC, zinfo.compress_size,
-              zinfo.file_size))
+        self.fp.seek(zinfo.header_offset, 0)
+        self.fp.write(zinfo.FileHeader(zip64))
         self.fp.seek(position, 0)
         self.filelist.append(zinfo)
         self.NameToInfo[zinfo.filename] = zinfo
@@ -1136,14 +1222,18 @@
             zinfo.compress_size = len(bytes)    # Compressed size
         else:
             zinfo.compress_size = zinfo.file_size
-        zinfo.header_offset = self.fp.tell()    # Start of header bytes
-        self.fp.write(zinfo.FileHeader())
+        zip64 = zinfo.file_size > ZIP64_LIMIT or \
+                zinfo.compress_size > ZIP64_LIMIT
+        if zip64 and not self._allowZip64:
+            raise LargeZipFile("Filesize would require ZIP64 extensions")
+        self.fp.write(zinfo.FileHeader(zip64))
         self.fp.write(bytes)
-        self.fp.flush()
         if zinfo.flag_bits & 0x08:
             # Write CRC and file sizes after the file data
-            self.fp.write(struct.pack("<LLL", zinfo.CRC, zinfo.compress_size,
+            fmt = '<LQQ' if zip64 else '<LLL'
+            self.fp.write(struct.pack(fmt, zinfo.CRC, zinfo.compress_size,
                   zinfo.file_size))
+        self.fp.flush()
         self.filelist.append(zinfo)
         self.NameToInfo[zinfo.filename] = zinfo
 
@@ -1157,109 +1247,104 @@
         if self.fp is None:
             return
 
-        if self.mode in ("w", "a") and self._didModify: # write ending records
-            count = 0
-            pos1 = self.fp.tell()
-            for zinfo in self.filelist:         # write central directory
-                count = count + 1
-                dt = zinfo.date_time
-                dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2]
-                dostime = dt[3] << 11 | dt[4] << 5 | (dt[5] // 2)
-                extra = []
-                if zinfo.file_size > ZIP64_LIMIT \
-                        or zinfo.compress_size > ZIP64_LIMIT:
-                    extra.append(zinfo.file_size)
-                    extra.append(zinfo.compress_size)
-                    file_size = 0xffffffff
-                    compress_size = 0xffffffff
-                else:
-                    file_size = zinfo.file_size
-                    compress_size = zinfo.compress_size
+        try:
+            if self.mode in ("w", "a") and self._didModify: # write ending records
+                count = 0
+                pos1 = self.fp.tell()
+                for zinfo in self.filelist:         # write central directory
+                    count = count + 1
+                    dt = zinfo.date_time
+                    dosdate = (dt[0] - 1980) << 9 | dt[1] << 5 | dt[2]
+                    dostime = dt[3] << 11 | dt[4] << 5 | (dt[5] // 2)
+                    extra = []
+                    if zinfo.file_size > ZIP64_LIMIT \
+                            or zinfo.compress_size > ZIP64_LIMIT:
+                        extra.append(zinfo.file_size)
+                        extra.append(zinfo.compress_size)
+                        file_size = 0xffffffff
+                        compress_size = 0xffffffff
+                    else:
+                        file_size = zinfo.file_size
+                        compress_size = zinfo.compress_size
 
-                if zinfo.header_offset > ZIP64_LIMIT:
-                    extra.append(zinfo.header_offset)
-                    header_offset = 0xffffffffL
-                else:
-                    header_offset = zinfo.header_offset
+                    if zinfo.header_offset > ZIP64_LIMIT:
+                        extra.append(zinfo.header_offset)
+                        header_offset = 0xffffffffL
+                    else:
+                        header_offset = zinfo.header_offset
 
-                extra_data = zinfo.extra
-                if extra:
-                    # Append a ZIP64 field to the extra's
-                    extra_data = struct.pack(
-                            '<HH' + 'Q'*len(extra),
-                            1, 8*len(extra), *extra) + extra_data
+                    extra_data = zinfo.extra
+                    if extra:
+                        # Append a ZIP64 field to the extra's
+                        extra_data = struct.pack(
+                                '<HH' + 'Q'*len(extra),
+                                1, 8*len(extra), *extra) + extra_data
 
-                    extract_version = max(45, zinfo.extract_version)
-                    create_version = max(45, zinfo.create_version)
-                else:
-                    extract_version = zinfo.extract_version
-                    create_version = zinfo.create_version
+                        extract_version = max(45, zinfo.extract_version)
+                        create_version = max(45, zinfo.create_version)
+                    else:
+                        extract_version = zinfo.extract_version
+                        create_version = zinfo.create_version
 
-                try:
-                    filename, flag_bits = zinfo._encodeFilenameFlags()
-                    centdir = struct.pack(structCentralDir,
-                     stringCentralDir, create_version,
-                     zinfo.create_system, extract_version, zinfo.reserved,
-                     flag_bits, zinfo.compress_type, dostime, dosdate,
-                     zinfo.CRC, compress_size, file_size,
-                     len(filename), len(extra_data), len(zinfo.comment),
-                     0, zinfo.internal_attr, zinfo.external_attr,
-                     header_offset)
-                except DeprecationWarning:
-                    print >>sys.stderr, (structCentralDir,
-                     stringCentralDir, create_version,
-                     zinfo.create_system, extract_version, zinfo.reserved,
-                     zinfo.flag_bits, zinfo.compress_type, dostime, dosdate,
-                     zinfo.CRC, compress_size, file_size,
-                     len(zinfo.filename), len(extra_data), len(zinfo.comment),
-                     0, zinfo.internal_attr, zinfo.external_attr,
-                     header_offset)
-                    raise
-                self.fp.write(centdir)
-                self.fp.write(filename)
-                self.fp.write(extra_data)
-                self.fp.write(zinfo.comment)
+                    try:
+                        filename, flag_bits = zinfo._encodeFilenameFlags()
+                        centdir = struct.pack(structCentralDir,
+                        stringCentralDir, create_version,
+                        zinfo.create_system, extract_version, zinfo.reserved,
+                        flag_bits, zinfo.compress_type, dostime, dosdate,
+                        zinfo.CRC, compress_size, file_size,
+                        len(filename), len(extra_data), len(zinfo.comment),
+                        0, zinfo.internal_attr, zinfo.external_attr,
+                        header_offset)
+                    except DeprecationWarning:
+                        print >>sys.stderr, (structCentralDir,
+                        stringCentralDir, create_version,
+                        zinfo.create_system, extract_version, zinfo.reserved,
+                        zinfo.flag_bits, zinfo.compress_type, dostime, dosdate,
+                        zinfo.CRC, compress_size, file_size,
+                        len(zinfo.filename), len(extra_data), len(zinfo.comment),
+                        0, zinfo.internal_attr, zinfo.external_attr,
+                        header_offset)
+                        raise
+                    self.fp.write(centdir)
+                    self.fp.write(filename)
+                    self.fp.write(extra_data)
+                    self.fp.write(zinfo.comment)
 
-            pos2 = self.fp.tell()
-            # Write end-of-zip-archive record
-            centDirCount = count
-            centDirSize = pos2 - pos1
-            centDirOffset = pos1
-            if (centDirCount >= ZIP_FILECOUNT_LIMIT or
-                centDirOffset > ZIP64_LIMIT or
-                centDirSize > ZIP64_LIMIT):
-                # Need to write the ZIP64 end-of-archive records
-                zip64endrec = struct.pack(
-                        structEndArchive64, stringEndArchive64,
-                        44, 45, 45, 0, 0, centDirCount, centDirCount,
-                        centDirSize, centDirOffset)
-                self.fp.write(zip64endrec)
+                pos2 = self.fp.tell()
+                # Write end-of-zip-archive record
+                centDirCount = count
+                centDirSize = pos2 - pos1
+                centDirOffset = pos1
+                if (centDirCount >= ZIP_FILECOUNT_LIMIT or
+                    centDirOffset > ZIP64_LIMIT or
+                    centDirSize > ZIP64_LIMIT):
+                    # Need to write the ZIP64 end-of-archive records
+                    zip64endrec = struct.pack(
+                            structEndArchive64, stringEndArchive64,
+                            44, 45, 45, 0, 0, centDirCount, centDirCount,
+                            centDirSize, centDirOffset)
+                    self.fp.write(zip64endrec)
 
-                zip64locrec = struct.pack(
-                        structEndArchive64Locator,
-                        stringEndArchive64Locator, 0, pos2, 1)
-                self.fp.write(zip64locrec)
-                centDirCount = min(centDirCount, 0xFFFF)
-                centDirSize = min(centDirSize, 0xFFFFFFFF)
-                centDirOffset = min(centDirOffset, 0xFFFFFFFF)
+                    zip64locrec = struct.pack(
+                            structEndArchive64Locator,
+                            stringEndArchive64Locator, 0, pos2, 1)
+                    self.fp.write(zip64locrec)
+                    centDirCount = min(centDirCount, 0xFFFF)
+                    centDirSize = min(centDirSize, 0xFFFFFFFF)
+                    centDirOffset = min(centDirOffset, 0xFFFFFFFF)
 
-            # check for valid comment length
-            if len(self.comment) >= ZIP_MAX_COMMENT:
-                if self.debug > 0:
-                    msg = 'Archive comment is too long; truncating to %d bytes' \
-                          % ZIP_MAX_COMMENT
-                self.comment = self.comment[:ZIP_MAX_COMMENT]
-
-            endrec = struct.pack(structEndArchive, stringEndArchive,
-                                 0, 0, centDirCount, centDirCount,
-                                 centDirSize, centDirOffset, len(self.comment))
-            self.fp.write(endrec)
-            self.fp.write(self.comment)
-            self.fp.flush()
-
-        if not self._filePassed:
-            self.fp.close()
-        self.fp = None
+                endrec = struct.pack(structEndArchive, stringEndArchive,
+                                    0, 0, centDirCount, centDirCount,
+                                    centDirSize, centDirOffset, len(self._comment))
+                self.fp.write(endrec)
+                self.fp.write(self._comment)
+                self.fp.flush()
+        finally:
+            fp = self.fp
+            self.fp = None
+            if not self._filePassed:
+                fp.close()
 
 
 class PyZipFile(ZipFile):
@@ -1338,7 +1423,7 @@
         /python/lib/string, return (/python/lib/string.pyc, string).
         """
         file_py  = pathname + ".py"
-        file_pyc = pathname + (".pyc" if not is_jython else "$py.class")
+        file_pyc = pathname + ".pyc"
         file_pyo = pathname + ".pyo"
         if os.path.isfile(file_pyo) and \
                             os.stat(file_pyo).st_mtime >= os.stat(file_py).st_mtime:
@@ -1381,16 +1466,15 @@
         if len(args) != 2:
             print USAGE
             sys.exit(1)
-        zf = ZipFile(args[1], 'r')
-        zf.printdir()
-        zf.close()
+        with ZipFile(args[1], 'r') as zf:
+            zf.printdir()
 
     elif args[0] == '-t':
         if len(args) != 2:
             print USAGE
             sys.exit(1)
-        zf = ZipFile(args[1], 'r')
-        badfile = zf.testzip()
+        with ZipFile(args[1], 'r') as zf:
+            badfile = zf.testzip()
         if badfile:
             print("The following enclosed file is corrupted: {!r}".format(badfile))
         print "Done testing"
@@ -1400,20 +1484,19 @@
             print USAGE
             sys.exit(1)
 
-        zf = ZipFile(args[1], 'r')
-        out = args[2]
-        for path in zf.namelist():
-            if path.startswith('./'):
-                tgt = os.path.join(out, path[2:])
-            else:
-                tgt = os.path.join(out, path)
+        with ZipFile(args[1], 'r') as zf:
+            out = args[2]
+            for path in zf.namelist():
+                if path.startswith('./'):
+                    tgt = os.path.join(out, path[2:])
+                else:
+                    tgt = os.path.join(out, path)
 
-            tgtdir = os.path.dirname(tgt)
-            if not os.path.exists(tgtdir):
-                os.makedirs(tgtdir)
-            with open(tgt, 'wb') as fp:
-                fp.write(zf.read(path))
-        zf.close()
+                tgtdir = os.path.dirname(tgt)
+                if not os.path.exists(tgtdir):
+                    os.makedirs(tgtdir)
+                with open(tgt, 'wb') as fp:
+                    fp.write(zf.read(path))
 
     elif args[0] == '-c':
         if len(args) < 3:
@@ -1429,11 +1512,9 @@
                             os.path.join(path, nm), os.path.join(zippath, nm))
             # else: ignore
 
-        zf = ZipFile(args[1], 'w', allowZip64=True)
-        for src in args[2:]:
-            addToZip(zf, src, os.path.basename(src))
-
-        zf.close()
+        with ZipFile(args[1], 'w', allowZip64=True) as zf:
+            for src in args[2:]:
+                addToZip(zf, src, os.path.basename(src))
 
 if __name__ == "__main__":
     main()

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 20 00:58:20 2015
From: jython-checkins at python.org (santoso.wijaya)
Date: Thu, 19 Mar 2015 23:58:20 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Use_position=28long=29_sett?=
 =?utf-8?q?er_to_manually_update_file_channel=27s_position_post-write=2E?=
Message-ID: <20150319235820.70040.61046@psf.io>

https://hg.python.org/jython/rev/e0f84f668a19
changeset:   7621:e0f84f668a19
user:        Santoso Wijaya <santoso.wijaya at gmail.com>
date:        Thu Mar 19 17:06:53 2015 -0700
summary:
  Use position(long) setter to manually update file channel's position post-write.

This fixes issue #2068.

files:
  Lib/test/test_os_jy.py             |  15 +++++++++++++++
  src/org/python/core/io/FileIO.java |  16 +++++++++++++---
  2 files changed, 28 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_os_jy.py b/Lib/test/test_os_jy.py
--- a/Lib/test/test_os_jy.py
+++ b/Lib/test/test_os_jy.py
@@ -59,6 +59,21 @@
             else:
                 self.assertTrue(False)
 
+    def test_issue2068(self):
+        os.remove(test_support.TESTFN)
+        for i in range(2):
+            fd = os.open(test_support.TESTFN, os.O_RDWR | os.O_CREAT | os.O_APPEND)
+            try:
+                os.write(fd, bytes('one'))
+                os.write(fd, bytes('two'))
+                os.write(fd, bytes('three'))
+            finally:
+                fd.close()
+
+        with open(test_support.TESTFN, 'rb') as f:
+            content = f.read()
+        self.assertEqual(content, 2 * b'onetwothree')
+
 
 class OSDirTestCase(unittest.TestCase):
 
diff --git a/src/org/python/core/io/FileIO.java b/src/org/python/core/io/FileIO.java
--- a/src/org/python/core/io/FileIO.java
+++ b/src/org/python/core/io/FileIO.java
@@ -306,13 +306,22 @@
         checkClosed();
         checkWritable();
         try {
-            return !emulateAppend ? fileChannel.write(buf) :
-                    fileChannel.write(buf, fileChannel.position());
+            return emulateAppend ? appendFromByteBuffer(buf)    // use this helper function to advance the file channel's position post-write
+                                    : fileChannel.write(buf);   // this does change the file channel's position
         } catch (IOException ioe) {
             throw Py.IOError(ioe);
         }
     }
 
+    private int appendFromByteBuffer(ByteBuffer buf) throws IOException {
+        int written = fileChannel.write(buf, fileChannel.position());   // this does not change the file channel's position!
+        if (written > 0) {
+            // we need to manually update the file channel's position post-write
+            fileChannel.position(fileChannel.position() + written);
+        }
+        return written;
+    }
+
     /**
      * Write bytes from each of the specified ByteBuffers via gather
      * i/o.
@@ -344,7 +353,8 @@
             if (!buf.hasRemaining()) {
                 continue;
             }
-            if ((bufCount = fileChannel.write(buf, fileChannel.position())) == 0) {
+            bufCount = appendFromByteBuffer(buf);
+            if (bufCount == 0) {
                 break;
             }
             count += bufCount;

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 20 19:44:16 2015
From: jython-checkins at python.org (jim.baker)
Date: Fri, 20 Mar 2015 18:44:16 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_More_robustness_wrt_securit?=
 =?utf-8?q?y_managers=2C_python=2Ehome=2C_import_of_non-ascii_names?=
Message-ID: <20150320184415.70036.16496@psf.io>

https://hg.python.org/jython/rev/b0d8aed3a8cd
changeset:   7622:b0d8aed3a8cd
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 20 12:44:04 2015 -0600
summary:
  More robustness wrt security managers, python.home, import of non-ascii names

Using a reasonable security manager profile (support for properties)
with Jython now works with the site and os modules.

Generates a diagnostic message when attempting to import site if this
module or its dependencies, such as posixpath, cannot be imported

Adds new Java property python.import.site, which can be set to false
to not import the site module.

Non ascii names no longer fail with j.l.IllegalArgumentException, but
instead raise UnicodeEncodeError. Note that there is no support for
byte string module names that are not ascii.

Fixes http://bugs.jython.org/issue1371 (testing security manager support),
http://bugs.jython.org/issue2283 (python.home and importing site),
http://bugs.jython.org/issue2296 (non ascii module names)

files:
  Lib/pwd.py                                    |   7 +-
  Lib/signal.py                                 |   4 +-
  Lib/site.py                                   |   7 +-
  Lib/sysconfig.py                              |  19 ++-
  Lib/test/test_import_jy.py                    |  19 ++-
  Lib/test/test_java_integration.py             |  19 ++-
  Lib/test/test_site_jy.py                      |  58 ++++++++++
  src/org/python/core/Options.java              |   2 +
  src/org/python/core/Py.java                   |  50 +++++++-
  src/org/python/core/PySystemState.java        |   3 +-
  src/org/python/core/PyUnicode.java            |   2 +-
  src/org/python/core/imp.java                  |   5 +
  src/org/python/modules/posix/PosixModule.java |  37 +++--
  src/org/python/util/PythonInterpreter.java    |   5 +-
  src/org/python/util/jython.java               |  21 +--
  15 files changed, 197 insertions(+), 61 deletions(-)


diff --git a/Lib/pwd.py b/Lib/pwd.py
--- a/Lib/pwd.py
+++ b/Lib/pwd.py
@@ -10,8 +10,11 @@
 
 __all__ = ['getpwuid', 'getpwnam', 'getpwall']
 
-from os import _name, _posix_impl
-from org.python.core.Py import newStringOrUnicode
+try:
+    from os import _name, _posix_impl
+    from org.python.core.Py import newStringOrUnicode
+except:
+    raise ImportError
 import sys
 
 if _name == 'nt':
diff --git a/Lib/signal.py b/Lib/signal.py
--- a/Lib/signal.py
+++ b/Lib/signal.py
@@ -29,11 +29,13 @@
     an ImportError is raised.
 """
 
-
+from java.lang import SecurityException
 try:
     import sun.misc.Signal
 except ImportError:
     raise ImportError("signal module requires sun.misc.Signal, which is not available on this platform")
+except SecurityException, ex:
+    raise ImportError("signal module requires sun.misc.Signal, which is not allowed by your security profile: %s" % ex)
 
 import os
 import sun.misc.SignalHandler
diff --git a/Lib/site.py b/Lib/site.py
--- a/Lib/site.py
+++ b/Lib/site.py
@@ -58,9 +58,14 @@
 
 """
 
+# these first two imports will always be available if we got this far,
+# since they are implemented in Java
+import __builtin__
 import sys
+
+# but the remaining imports may fail, if so, this is because
+# python.home is incorrect; see the diagnosis in Py.importSiteIfSelected
 import os
-import __builtin__
 import traceback
 
 _is_jython = sys.platform.startswith("java")
diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py
--- a/Lib/sysconfig.py
+++ b/Lib/sysconfig.py
@@ -109,8 +109,8 @@
 _PY_VERSION = sys.version.split()[0]
 _PY_VERSION_SHORT = sys.version[:3]
 _PY_VERSION_SHORT_NO_DOT = _PY_VERSION[0] + _PY_VERSION[2]
-_PREFIX = os.path.normpath(sys.prefix)
-_EXEC_PREFIX = os.path.normpath(sys.exec_prefix)
+_PREFIX = os.path.normpath(sys.prefix) if sys.prefix is not None else None
+_EXEC_PREFIX = os.path.normpath(sys.exec_prefix) if sys.exec_prefix is not None else None
 _CONFIG_VARS = None
 _USER_BASE = None
 
@@ -173,7 +173,10 @@
 
     for key, value in _INSTALL_SCHEMES[scheme].items():
         if os.name in ('posix', 'nt', 'java'):
-            value = os.path.expanduser(value)
+            try:
+                value = os.path.expanduser(value)
+            except ImportError:
+                pass  # ignore missing pwd if no native posix for Jython
         res[key] = os.path.normpath(_subst_vars(value, vars))
     return res
 
@@ -189,7 +192,7 @@
         return os.path.expanduser(os.path.join(*args))
 
     # what about 'os2emx', 'riscos' ?
-    if os.name == "nt":
+    if os.name == "nt" or os._name == "nt":
         base = os.environ.get("APPDATA") or "~"
         return env_base if env_base else joinuser(base, "Python")
 
@@ -199,8 +202,12 @@
             return env_base if env_base else \
                                joinuser("~", "Library", framework, "%d.%d"
                                             % (sys.version_info[:2]))
-
-    return env_base if env_base else joinuser("~", ".local")
+    if env_base:
+        return env_base
+    try:
+        return joinuser("~", ".local")
+    except:
+        return None  # SecurityManager prevents this for Jython
 
 
 def _parse_makefile(filename, vars=None):
diff --git a/Lib/test/test_import_jy.py b/Lib/test/test_import_jy.py
--- a/Lib/test/test_import_jy.py
+++ b/Lib/test/test_import_jy.py
@@ -1,3 +1,4 @@
+# -*- coding: utf-8 -*-
 """Misc. import tests
 
 Made for Jython.
@@ -194,9 +195,8 @@
             try:
                 os.mkdir(test_support.TESTFN)
                 init = os.path.join(test_support.TESTFN, "__init__.py")
-                fp = open(init, 'w')
-                fp.write("test = 'imported'")
-                fp.close()
+                with open(init, 'w') as fp:
+                    fp.write("test = 'imported'")
                 os.symlink(test_support.TESTFN, sym)
                 module = os.path.basename(sym)
                 module_obj = __import__(module)
@@ -217,10 +217,21 @@
         __import__("os", [], level=-1)
 
 
+class UnicodeNamesTestCase(unittest.TestCase):
+
+    def test_import_unicode_module(self):
+        with self.assertRaises(UnicodeEncodeError) as cm:
+            __import__("m?d?l?")
+        self.assertEqual(cm.exception.encoding, "ascii")
+        self.assertEqual(cm.exception.object, "m?d?l?")
+        self.assertEqual(cm.exception.reason, "ordinal not in range(128)")
+
+
 def test_main():
     test_support.run_unittest(MislabeledImportTestCase,
                               OverrideBuiltinsImportTestCase,
-                              ImpTestCase)
+                              ImpTestCase,
+                              UnicodeNamesTestCase)
 
 if __name__ == '__main__':
     test_main()
diff --git a/Lib/test/test_java_integration.py b/Lib/test/test_java_integration.py
--- a/Lib/test/test_java_integration.py
+++ b/Lib/test/test_java_integration.py
@@ -476,11 +476,7 @@
 
 class SecurityManagerTest(unittest.TestCase):
 
-    @unittest.skip("XXX: not working")
     def test_nonexistent_import_with_security(self):
-        if os._name == 'nt':
-            # http://bugs.jython.org/issue1371
-            return
         script = test_support.findfile("import_nonexistent.py")
         home = os.path.realpath(sys.prefix)
         if not os.path.commonprefix((home, os.path.realpath(script))) == home:
@@ -491,6 +487,21 @@
             "-J-Djava.security.manager", "-J-Djava.security.policy=%s" % policy, script]),
             0)
 
+    def test_import_signal_fails_with_import_error_using_security(self):
+        policy = test_support.findfile("python_home.policy")
+        with self.assertRaises(subprocess.CalledProcessError) as cm:
+            subprocess.check_output(
+                [sys.executable,
+                 "-J-Dpython.cachedir.skip=true",
+                 "-J-Djava.security.manager",
+                 "-J-Djava.security.policy=%s" % policy,
+                 "-c", "import signal"],
+                stderr=subprocess.STDOUT)
+        self.assertIn(
+            'ImportError: signal module requires sun.misc.Signal, which is not allowed by your security profile',
+            cm.exception.output)
+
+
 class JavaWrapperCustomizationTest(unittest.TestCase):
     def tearDown(self):
         CustomizableMapHolder.clearAdditions()
diff --git a/Lib/test/test_site_jy.py b/Lib/test/test_site_jy.py
new file mode 100644
--- /dev/null
+++ b/Lib/test/test_site_jy.py
@@ -0,0 +1,58 @@
+import os
+import subprocess
+import sys
+import unittest
+from test import test_support
+
+
+class ImportSiteTestCase(unittest.TestCase):
+    
+    def test_empty_python_home(self):
+        # http://bugs.jython.org/issue2283
+        with test_support.temp_cwd() as temp_cwd:
+            # using a new directory ensures no Lib/ directory is available
+            self.assertEqual(
+                subprocess.check_output(
+                    [sys.executable, "-Dpython.home=", "-c",
+                     "import os; os.system('echo 42'); os.system('echo 47')"])\
+                .replace("\r", ""),  # in case of running on Windows
+                "42\n47\n")
+
+    def test_bad_python_home(self):
+        # http://bugs.jython.org/issue2283
+        with test_support.temp_cwd() as temp_cwd:
+            os.makedirs(os.path.join(temp_cwd, "Lib"))
+            with self.assertRaises(subprocess.CalledProcessError) as cm:
+                subprocess.check_output(
+                    [sys.executable, "-Dpython.home=%s" % temp_cwd, "-c",
+                     "print 42"],
+                    stderr=subprocess.STDOUT)
+            self.assertIn(
+                'Exception in thread "main" ImportError: Cannot import site module and its dependencies: No module named site',
+                cm.exception.output)
+
+    def test_property_no_site_import(self):
+        # only the minimal set of modules are imported
+        with test_support.temp_cwd() as temp_cwd:
+            self.assertEqual(
+                subprocess.check_output(
+                    [sys.executable, "-Dpython.import.site=false", "-c",
+                     "import sys; print sorted(sys.modules.keys())"]).strip(),
+                "['__builtin__', '__main__', 'exceptions', 'sys']")
+
+    def test_options_no_site_import(self):
+        with test_support.temp_cwd() as temp_cwd:
+            self.assertEqual(
+                subprocess.check_output(
+                    [sys.executable, "-S", "-c",
+                     "import sys; print sorted(sys.modules.keys())"]).strip(),
+                "['__builtin__', '__main__', 'exceptions', 'sys']")
+
+
+def test_main():
+    test_support.run_unittest(
+        ImportSiteTestCase,
+    )
+
+if __name__ == '__main__':
+    test_main()
diff --git a/src/org/python/core/Options.java b/src/org/python/core/Options.java
--- a/src/org/python/core/Options.java
+++ b/src/org/python/core/Options.java
@@ -201,5 +201,7 @@
         }
 
         Options.sreCacheSpec = getStringOption("sre.cachespec", Options.sreCacheSpec);
+
+        Options.importSite = getBooleanOption("import.site", Options.importSite);
     }
 }
diff --git a/src/org/python/core/Py.java b/src/org/python/core/Py.java
--- a/src/org/python/core/Py.java
+++ b/src/org/python/core/Py.java
@@ -999,12 +999,11 @@
             return;
         }
 
-        if (Options.importSite) {
-            // Ensure site-packages are available before attempting to import module.
-            // This step enables supporting modern Python apps when using proxies
-            // directly from Java (eg through clamp).
-            imp.load("site");
-        }
+        // Ensure site-packages are available before attempting to import module.
+        // This step enables supporting modern Python apps when using proxies
+        // directly from Java (eg through clamp).
+        importSiteIfSelected();
+
         PyObject mod = imp.importName(module.intern(), false);
         PyType pyc = (PyType)mod.__getattr__(pyclass.intern());
 
@@ -1530,9 +1529,42 @@
             return true;
         }
         // Decide if System.in is interactive
-        POSIX posix = POSIXFactory.getPOSIX();
-        FileDescriptor in = FileDescriptor.in;
-        return posix.isatty(in);
+        try {
+            POSIX posix = POSIXFactory.getPOSIX();
+            FileDescriptor in = FileDescriptor.in;
+            return posix.isatty(in);
+        } catch (SecurityException ex) {
+            return false;
+        }
+    }
+
+    public static boolean importSiteIfSelected() {
+        if (Options.importSite) {
+            try {
+                // Ensure site-packages are available
+                imp.load("site");
+                return true;
+            } catch (PyException pye) {
+                if (pye.match(Py.ImportError)) {
+                    PySystemState sys = Py.getSystemState();
+                    throw Py.ImportError(String.format(""
+                                    + "Cannot import site module and its dependencies: %s\n"
+                                    + "Determine if the following attributes are correct:\n"
+                                    + "  * sys.path: %s\n"
+                                    + "    This attribute might be including the wrong directories, such as from CPython\n"
+                                    + "  * sys.prefix: %s\n"
+                                    + "    This attribute is set by the system property python.home, although it can\n"
+                                    + "    be often automatically determined by the location of the Jython jar file\n\n"
+                                    + "You can use the -S option or python.import.site=false to not import the site module",
+                            pye.value.__getattr__("args").__getitem__(0),
+                            sys.path,
+                            sys.prefix));
+                } else {
+                    throw pye;
+                }
+            }
+        }
+        return false;
     }
 
     /* A collection of functions for implementing the print statement */
diff --git a/src/org/python/core/PySystemState.java b/src/org/python/core/PySystemState.java
--- a/src/org/python/core/PySystemState.java
+++ b/src/org/python/core/PySystemState.java
@@ -718,13 +718,12 @@
             if (root == null) {
                 root = preProperties.getProperty("install.root");
             }
-
             determinePlatform(preProperties);
         } catch (Exception exc) {
             return null;
         }
         // If install.root is undefined find JYTHON_JAR in class.path
-        if (root == null) {
+        if (root == null || root.equals("")) {
             String classpath = preProperties.getProperty("java.class.path");
             if (classpath != null) {
                 String lowerCaseClasspath = classpath.toLowerCase();
diff --git a/src/org/python/core/PyUnicode.java b/src/org/python/core/PyUnicode.java
--- a/src/org/python/core/PyUnicode.java
+++ b/src/org/python/core/PyUnicode.java
@@ -581,7 +581,7 @@
         return string.length() - translator.suppCount();
     }
 
-    private static String checkEncoding(String s) {
+    public static String checkEncoding(String s) {
         if (s == null || CharMatcher.ASCII.matchesAllOf(s)) { return s; }
         return codecs.PyUnicode_EncodeASCII(s, s.length(), null);
     }
diff --git a/src/org/python/core/imp.java b/src/org/python/core/imp.java
--- a/src/org/python/core/imp.java
+++ b/src/org/python/core/imp.java
@@ -413,6 +413,7 @@
      * is a jar moduleLocation should be the full uri for c.
      */
     public static PyObject createFromCode(String name, PyCode c, String moduleLocation) {
+        PyUnicode.checkEncoding(name);
         PyModule module = addModule(name);
 
         PyTableCode code = null;
@@ -578,6 +579,7 @@
     }
 
     static PyObject loadFromLoader(PyObject importer, String name) {
+        PyUnicode.checkEncoding(name);
         PyObject load_module = importer.__getattr__("load_module");
         ReentrantLock importLock = Py.getSystemState().getImportLock();
         importLock.lock();
@@ -707,6 +709,7 @@
      * @return the loaded module
      */
     public static PyObject load(String name) {
+        PyUnicode.checkEncoding(name);
         ReentrantLock importLock = Py.getSystemState().getImportLock();
         importLock.lock();
         try {
@@ -1031,6 +1034,7 @@
      * @return an imported module (Java or Python)
      */
     public static PyObject importName(String name, boolean top) {
+        PyUnicode.checkEncoding(name);
         ReentrantLock importLock = Py.getSystemState().getImportLock();
         importLock.lock();
         try {
@@ -1051,6 +1055,7 @@
      */
     public static PyObject importName(String name, boolean top,
             PyObject modDict, PyObject fromlist, int level) {
+        PyUnicode.checkEncoding(name);
         ReentrantLock importLock = Py.getSystemState().getImportLock();
         importLock.lock();
         try {
diff --git a/src/org/python/modules/posix/PosixModule.java b/src/org/python/modules/posix/PosixModule.java
--- a/src/org/python/modules/posix/PosixModule.java
+++ b/src/org/python/modules/posix/PosixModule.java
@@ -114,9 +114,15 @@
         // Successful termination
         dict.__setitem__("EX_OK", Py.Zero);
 
-        boolean nativePosix = posix.isNative();
-        dict.__setitem__("_native_posix", Py.newBoolean(nativePosix));
-        dict.__setitem__("_posix_impl", Py.java2py(posix));
+        // SecurityManager may restrict access to native implementation,
+        // so use Java-only implementation as necessary
+        boolean nativePosix = false;
+        try {
+            nativePosix = posix.isNative();
+            dict.__setitem__("_native_posix", Py.newBoolean(nativePosix));
+            dict.__setitem__("_posix_impl", Py.java2py(posix));
+        } catch (SecurityException ex) {}
+
         dict.__setitem__("environ", getEnviron());
         dict.__setitem__("error", Py.OSError);
         dict.__setitem__("stat_result", PyStatResult.TYPE);
@@ -483,7 +489,7 @@
     public static PyString __doc__getgid = new PyString(
         "getgid() -> gid\n\n" +
         "Return the current process's group id.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static int getgid() {
         return posix.getgid();
     }
@@ -491,7 +497,7 @@
     public static PyString __doc__getlogin = new PyString(
         "getlogin() -> string\n\n" +
         "Return the actual login name.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static PyObject getlogin() {
         return new PyString(posix.getlogin());
     }
@@ -499,7 +505,7 @@
     public static PyString __doc__getppid = new PyString(
         "getppid() -> ppid\n\n" +
         "Return the parent's process id.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static int getppid() {
         return posix.getppid();
     }
@@ -507,7 +513,7 @@
     public static PyString __doc__getuid = new PyString(
         "getuid() -> uid\n\n" +
         "Return the current process's user id.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static int getuid() {
         return posix.getuid();
     }
@@ -524,7 +530,7 @@
     public static PyString __doc__getpgrp = new PyString(
         "getpgrp() -> pgrp\n\n" +
         "Return the current process group id.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static int getpgrp() {
         return posix.getpgrp();
     }
@@ -535,6 +541,7 @@
         "isatty(fd) -> bool\n\n" +
         "Return True if the file descriptor 'fd' is an open file descriptor\n" +
         "connected to the slave end of a terminal.");
+    @Hide(posixImpl = PosixImpl.JAVA)
     public static boolean isatty(PyObject fdObj) {
         Object tojava = fdObj.__tojava__(IOBase.class);
         if (tojava != Py.NoConversion) {
@@ -568,7 +575,7 @@
     public static PyString __doc__kill = new PyString(
         "kill(pid, sig)\n\n" +
         "Kill a process with a signal.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static void kill(int pid, int sig) {
         if (posix.kill(pid, sig) < 0) {
             throw errorFromErrno();
@@ -579,7 +586,7 @@
         "lchmod(path, mode)\n\n" +
         "Change the access permissions of a file. If path is a symlink, this\n" +
         "affects the link itself rather than the target.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static void lchmod(PyObject path, int mode) {
         if (posix.lchmod(absolutePath(path).toString(), mode) < 0) {
             throw errorFromErrno(path);
@@ -590,7 +597,7 @@
         "lchown(path, uid, gid)\n\n" +
         "Change the owner and group id of path to the numeric uid and gid.\n" +
         "This function will not follow symbolic links.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static void lchown(PyObject path, int uid, int gid) {
         if (posix.lchown(absolutePath(path).toString(), uid, gid) < 0) {
             throw errorFromErrno(path);
@@ -842,7 +849,7 @@
     public static PyString __doc__setpgrp = new PyString(
         "setpgrp()\n\n" +
         "Make this process a session leader.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static void setpgrp() {
         if (posix.setpgrp(0, 0) < 0) {
             throw errorFromErrno();
@@ -852,7 +859,7 @@
     public static PyString __doc__setsid = new PyString(
         "setsid()\n\n" +
         "Call the system call setsid().");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static void setsid() {
         if (posix.setsid() < 0) {
             throw errorFromErrno();
@@ -901,6 +908,7 @@
         "times() -> (utime, stime, cutime, cstime, elapsed_time)\n\n" +
         "Return a tuple of floating point numbers indicating process times.");
 
+    @Hide(posixImpl = PosixImpl.JAVA)
     public static PyTuple times() {
         Times times = posix.times();
         long CLK_TCK = Sysconf._SC_CLK_TCK.longValue();
@@ -991,7 +999,7 @@
     public static PyString __doc__wait = new PyString(
         "wait() -> (pid, status)\n\n" +
         "Wait for completion of a child process.");
-    @Hide(OS.NT)
+    @Hide(value=OS.NT, posixImpl = PosixImpl.JAVA)
     public static PyObject wait$() {
         int[] status = new int[1];
         int pid = posix.wait(status);
@@ -1004,6 +1012,7 @@
     public static PyString __doc__waitpid = new PyString(
         "wait() -> (pid, status)\n\n" +
         "Wait for completion of a child process.");
+    @Hide(posixImpl = PosixImpl.JAVA)
     public static PyObject waitpid(int pid, int options) {
         int[] status = new int[1];
         pid = posix.waitpid(pid, status, options);
diff --git a/src/org/python/util/PythonInterpreter.java b/src/org/python/util/PythonInterpreter.java
--- a/src/org/python/util/PythonInterpreter.java
+++ b/src/org/python/util/PythonInterpreter.java
@@ -113,10 +113,7 @@
             systemState.modules.__setitem__("__main__", module);
         }
 
-        if (Options.importSite) {
-            // Ensure site-packages are available
-            imp.load("site");
-        }
+        Py.importSiteIfSelected();
     }
 
     public PySystemState getSystemState() {
diff --git a/src/org/python/util/jython.java b/src/org/python/util/jython.java
--- a/src/org/python/util/jython.java
+++ b/src/org/python/util/jython.java
@@ -288,18 +288,9 @@
             System.err.println(InteractiveConsole.getDefaultBanner());
         }
 
-        if (Options.importSite) {
-            try {
-                imp.load("site");
-                if (opts.interactive && opts.notice && !opts.runModule) {
-                    System.err.println(COPYRIGHT);
-                }
-            } catch (PyException pye) {
-                if (!pye.match(Py.ImportError)) {
-                    System.err.println("error importing site");
-                    Py.printException(pye);
-                    System.exit(-1);
-                }
+        if (Py.importSiteIfSelected()) {
+            if (opts.interactive && opts.notice && !opts.runModule) {
+                System.err.println(COPYRIGHT);
             }
         }
 
@@ -359,7 +350,11 @@
                         throw Py.IOError(e);
                     }
                     try {
-                        if (PosixModule.getPOSIX().isatty(file.getFD())) {
+                        boolean isInteractive = false;
+                        try {
+                            isInteractive = PosixModule.getPOSIX().isatty(file.getFD());
+                        } catch (SecurityException ex) {}
+                        if (isInteractive) {
                             opts.interactive = true;
                             interp.interact(null, new PyFile(file));
                             return;

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 20 19:45:32 2015
From: jython-checkins at python.org (jim.baker)
Date: Fri, 20 Mar 2015 18:45:32 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Use_standard_CPython_versio?=
	=?utf-8?q?n_of_gettext_module?=
Message-ID: <20150320184532.63576.36640@psf.io>

https://hg.python.org/jython/rev/10ffb89911b5
changeset:   7623:10ffb89911b5
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 20 12:45:22 2015 -0600
summary:
  Use standard CPython version of gettext module

files:
  Lib/gettext.py |  592 -------------------------------------
  1 files changed, 0 insertions(+), 592 deletions(-)


diff --git a/Lib/gettext.py b/Lib/gettext.py
deleted file mode 100644
--- a/Lib/gettext.py
+++ /dev/null
@@ -1,592 +0,0 @@
-"""Internationalization and localization support.
-
-This module provides internationalization (I18N) and localization (L10N)
-support for your Python programs by providing an interface to the GNU gettext
-message catalog library.
-
-I18N refers to the operation by which a program is made aware of multiple
-languages.  L10N refers to the adaptation of your program, once
-internationalized, to the local language and cultural habits.
-
-"""
-
-# This module represents the integration of work, contributions, feedback, and
-# suggestions from the following people:
-#
-# Martin von Loewis, who wrote the initial implementation of the underlying
-# C-based libintlmodule (later renamed _gettext), along with a skeletal
-# gettext.py implementation.
-#
-# Peter Funk, who wrote fintl.py, a fairly complete wrapper around intlmodule,
-# which also included a pure-Python implementation to read .mo files if
-# intlmodule wasn't available.
-#
-# James Henstridge, who also wrote a gettext.py module, which has some
-# interesting, but currently unsupported experimental features: the notion of
-# a Catalog class and instances, and the ability to add to a catalog file via
-# a Python API.
-#
-# Barry Warsaw integrated these modules, wrote the .install() API and code,
-# and conformed all C and Python code to Python's coding standards.
-#
-# Francois Pinard and Marc-Andre Lemburg also contributed valuably to this
-# module.
-#
-# J. David Ibanez implemented plural forms. Bruno Haible fixed some bugs.
-#
-# TODO:
-# - Lazy loading of .mo files.  Currently the entire catalog is loaded into
-#   memory, but that's probably bad for large translated programs.  Instead,
-#   the lexical sort of original strings in GNU .mo files should be exploited
-#   to do binary searches and lazy initializations.  Or you might want to use
-#   the undocumented double-hash algorithm for .mo files with hash tables, but
-#   you'll need to study the GNU gettext code to do this.
-#
-# - Support Solaris .mo file formats.  Unfortunately, we've been unable to
-#   find this format documented anywhere.
-
-
-from __future__ import with_statement
-import locale, copy, os, re, struct, sys
-from errno import ENOENT
-
-
-__all__ = ['NullTranslations', 'GNUTranslations', 'Catalog',
-           'find', 'translation', 'install', 'textdomain', 'bindtextdomain',
-           'dgettext', 'dngettext', 'gettext', 'ngettext',
-           ]
-
-_default_localedir = os.path.join(sys.prefix, 'share', 'locale')
-
-
-def test(condition, true, false):
-    """
-    Implements the C expression:
-
-      condition ? true : false
-
-    Required to correctly interpret plural forms.
-    """
-    if condition:
-        return true
-    else:
-        return false
-
-
-def c2py(plural):
-    """Gets a C expression as used in PO files for plural forms and returns a
-    Python lambda function that implements an equivalent expression.
-    """
-    # Security check, allow only the "n" identifier
-    try:
-        from cStringIO import StringIO
-    except ImportError:
-        from StringIO import StringIO
-    import token, tokenize
-    tokens = tokenize.generate_tokens(StringIO(plural).readline)
-    try:
-        danger = [x for x in tokens if x[0] == token.NAME and x[1] != 'n']
-    except tokenize.TokenError:
-        raise ValueError, \
-              'plural forms expression error, maybe unbalanced parenthesis'
-    else:
-        if danger:
-            raise ValueError, 'plural forms expression could be dangerous'
-
-    # Replace some C operators by their Python equivalents
-    plural = plural.replace('&&', ' and ')
-    plural = plural.replace('||', ' or ')
-
-    expr = re.compile(r'\!([^=])')
-    plural = expr.sub(' not \\1', plural)
-
-    # Regular expression and replacement function used to transform
-    # "a?b:c" to "test(a,b,c)".
-    expr = re.compile(r'(.*?)\?(.*?):(.*)')
-    def repl(x):
-        return "test(%s, %s, %s)" % (x.group(1), x.group(2),
-                                     expr.sub(repl, x.group(3)))
-
-    # Code to transform the plural expression, taking care of parentheses
-    stack = ['']
-    for c in plural:
-        if c == '(':
-            stack.append('')
-        elif c == ')':
-            if len(stack) == 1:
-                # Actually, we never reach this code, because unbalanced
-                # parentheses get caught in the security check at the
-                # beginning.
-                raise ValueError, 'unbalanced parenthesis in plural form'
-            s = expr.sub(repl, stack.pop())
-            stack[-1] += '(%s)' % s
-        else:
-            stack[-1] += c
-    plural = expr.sub(repl, stack.pop())
-
-    return eval('lambda n: int(%s)' % plural)
-
-
-
-def _expand_lang(locale):
-    from locale import normalize
-    locale = normalize(locale)
-    COMPONENT_CODESET   = 1 << 0
-    COMPONENT_TERRITORY = 1 << 1
-    COMPONENT_MODIFIER  = 1 << 2
-    # split up the locale into its base components
-    mask = 0
-    pos = locale.find('@')
-    if pos >= 0:
-        modifier = locale[pos:]
-        locale = locale[:pos]
-        mask |= COMPONENT_MODIFIER
-    else:
-        modifier = ''
-    pos = locale.find('.')
-    if pos >= 0:
-        codeset = locale[pos:]
-        locale = locale[:pos]
-        mask |= COMPONENT_CODESET
-    else:
-        codeset = ''
-    pos = locale.find('_')
-    if pos >= 0:
-        territory = locale[pos:]
-        locale = locale[:pos]
-        mask |= COMPONENT_TERRITORY
-    else:
-        territory = ''
-    language = locale
-    ret = []
-    for i in range(mask+1):
-        if not (i & ~mask):  # if all components for this combo exist ...
-            val = language
-            if i & COMPONENT_TERRITORY: val += territory
-            if i & COMPONENT_CODESET:   val += codeset
-            if i & COMPONENT_MODIFIER:  val += modifier
-            ret.append(val)
-    ret.reverse()
-    return ret
-
-
-
-class NullTranslations:
-    def __init__(self, fp=None):
-        self._info = {}
-        self._charset = None
-        self._output_charset = None
-        self._fallback = None
-        if fp is not None:
-            self._parse(fp)
-
-    def _parse(self, fp):
-        pass
-
-    def add_fallback(self, fallback):
-        if self._fallback:
-            self._fallback.add_fallback(fallback)
-        else:
-            self._fallback = fallback
-
-    def gettext(self, message):
-        if self._fallback:
-            return self._fallback.gettext(message)
-        return message
-
-    def lgettext(self, message):
-        if self._fallback:
-            return self._fallback.lgettext(message)
-        return message
-
-    def ngettext(self, msgid1, msgid2, n):
-        if self._fallback:
-            return self._fallback.ngettext(msgid1, msgid2, n)
-        if n == 1:
-            return msgid1
-        else:
-            return msgid2
-
-    def lngettext(self, msgid1, msgid2, n):
-        if self._fallback:
-            return self._fallback.lngettext(msgid1, msgid2, n)
-        if n == 1:
-            return msgid1
-        else:
-            return msgid2
-
-    def ugettext(self, message):
-        if self._fallback:
-            return self._fallback.ugettext(message)
-        return unicode(message)
-
-    def ungettext(self, msgid1, msgid2, n):
-        if self._fallback:
-            return self._fallback.ungettext(msgid1, msgid2, n)
-        if n == 1:
-            return unicode(msgid1)
-        else:
-            return unicode(msgid2)
-
-    def info(self):
-        return self._info
-
-    def charset(self):
-        return self._charset
-
-    def output_charset(self):
-        return self._output_charset
-
-    def set_output_charset(self, charset):
-        self._output_charset = charset
-
-    def install(self, unicode=False, names=None):
-        import __builtin__
-        __builtin__.__dict__['_'] = unicode and self.ugettext or self.gettext
-        if hasattr(names, "__contains__"):
-            if "gettext" in names:
-                __builtin__.__dict__['gettext'] = __builtin__.__dict__['_']
-            if "ngettext" in names:
-                __builtin__.__dict__['ngettext'] = (unicode and self.ungettext
-                                                             or self.ngettext)
-            if "lgettext" in names:
-                __builtin__.__dict__['lgettext'] = self.lgettext
-            if "lngettext" in names:
-                __builtin__.__dict__['lngettext'] = self.lngettext
-
-
-class GNUTranslations(NullTranslations):
-    # Magic number of .mo files
-    LE_MAGIC = 0x950412deL
-    BE_MAGIC = 0xde120495L
-
-    def _parse(self, fp):
-        """Override this method to support alternative .mo formats."""
-        unpack = struct.unpack
-        filename = getattr(fp, 'name', '')
-        # Parse the .mo file header, which consists of 5 little endian 32
-        # bit words.
-        self._catalog = catalog = {}
-        self.plural = lambda n: int(n != 1) # germanic plural by default
-        buf = fp.read()
-        buflen = len(buf)
-        # Are we big endian or little endian?
-        magic = unpack('<I', buf[:4])[0]
-        if magic == self.LE_MAGIC:
-            version, msgcount, masteridx, transidx = unpack('<4I', buf[4:20])
-            ii = '<II'
-        elif magic == self.BE_MAGIC:
-            version, msgcount, masteridx, transidx = unpack('>4I', buf[4:20])
-            ii = '>II'
-        else:
-            raise IOError(0, 'Bad magic number', filename)
-        # Now put all messages from the .mo file buffer into the catalog
-        # dictionary.
-        for i in xrange(0, msgcount):
-            mlen, moff = unpack(ii, buf[masteridx:masteridx+8])
-            mend = moff + mlen
-            tlen, toff = unpack(ii, buf[transidx:transidx+8])
-            tend = toff + tlen
-            if mend < buflen and tend < buflen:
-                msg = buf[moff:mend]
-                tmsg = buf[toff:tend]
-            else:
-                raise IOError(0, 'File is corrupt', filename)
-            # See if we're looking at GNU .mo conventions for metadata
-            if mlen == 0:
-                # Catalog description
-                lastk = k = None
-                for item in tmsg.splitlines():
-                    item = item.strip()
-                    if not item:
-                        continue
-                    if ':' in item:
-                        k, v = item.split(':', 1)
-                        k = k.strip().lower()
-                        v = v.strip()
-                        self._info[k] = v
-                        lastk = k
-                    elif lastk:
-                        self._info[lastk] += '\n' + item
-                    if k == 'content-type':
-                        self._charset = v.split('charset=')[1]
-                    elif k == 'plural-forms':
-                        v = v.split(';')
-                        plural = v[1].split('plural=')[1]
-                        self.plural = c2py(plural)
-            # Note: we unconditionally convert both msgids and msgstrs to
-            # Unicode using the character encoding specified in the charset
-            # parameter of the Content-Type header.  The gettext documentation
-            # strongly encourages msgids to be us-ascii, but some appliations
-            # require alternative encodings (e.g. Zope's ZCML and ZPT).  For
-            # traditional gettext applications, the msgid conversion will
-            # cause no problems since us-ascii should always be a subset of
-            # the charset encoding.  We may want to fall back to 8-bit msgids
-            # if the Unicode conversion fails.
-            if '\x00' in msg:
-                # Plural forms
-                msgid1, msgid2 = msg.split('\x00')
-                tmsg = tmsg.split('\x00')
-                if self._charset:
-                    msgid1 = unicode(msgid1, self._charset)
-                    tmsg = [unicode(x, self._charset) for x in tmsg]
-                for i in range(len(tmsg)):
-                    catalog[(msgid1, i)] = tmsg[i]
-            else:
-                if self._charset:
-                    msg = unicode(msg, self._charset)
-                    tmsg = unicode(tmsg, self._charset)
-                catalog[msg] = tmsg
-            # advance to next entry in the seek tables
-            masteridx += 8
-            transidx += 8
-
-    def gettext(self, message):
-        missing = object()
-        tmsg = self._catalog.get(message, missing)
-        if tmsg is missing:
-            if self._fallback:
-                return self._fallback.gettext(message)
-            return message
-        # Encode the Unicode tmsg back to an 8-bit string, if possible
-        if self._output_charset:
-            return tmsg.encode(self._output_charset)
-        elif self._charset:
-            return tmsg.encode(self._charset)
-        return tmsg
-
-    def lgettext(self, message):
-        missing = object()
-        tmsg = self._catalog.get(message, missing)
-        if tmsg is missing:
-            if self._fallback:
-                return self._fallback.lgettext(message)
-            return message
-        if self._output_charset:
-            return tmsg.encode(self._output_charset)
-        return tmsg.encode(locale.getpreferredencoding())
-
-    def ngettext(self, msgid1, msgid2, n):
-        try:
-            tmsg = self._catalog[(msgid1, self.plural(n))]
-            if self._output_charset:
-                return tmsg.encode(self._output_charset)
-            elif self._charset:
-                return tmsg.encode(self._charset)
-            return tmsg
-        except KeyError:
-            if self._fallback:
-                return self._fallback.ngettext(msgid1, msgid2, n)
-            if n == 1:
-                return msgid1
-            else:
-                return msgid2
-
-    def lngettext(self, msgid1, msgid2, n):
-        try:
-            tmsg = self._catalog[(msgid1, self.plural(n))]
-            if self._output_charset:
-                return tmsg.encode(self._output_charset)
-            return tmsg.encode(locale.getpreferredencoding())
-        except KeyError:
-            if self._fallback:
-                return self._fallback.lngettext(msgid1, msgid2, n)
-            if n == 1:
-                return msgid1
-            else:
-                return msgid2
-
-    def ugettext(self, message):
-        missing = object()
-        tmsg = self._catalog.get(message, missing)
-        if tmsg is missing:
-            if self._fallback:
-                return self._fallback.ugettext(message)
-            return unicode(message)
-        return tmsg
-
-    def ungettext(self, msgid1, msgid2, n):
-        try:
-            tmsg = self._catalog[(msgid1, self.plural(n))]
-        except KeyError:
-            if self._fallback:
-                return self._fallback.ungettext(msgid1, msgid2, n)
-            if n == 1:
-                tmsg = unicode(msgid1)
-            else:
-                tmsg = unicode(msgid2)
-        return tmsg
-
-
-# Locate a .mo file using the gettext strategy
-def find(domain, localedir=None, languages=None, all=0):
-    # Get some reasonable defaults for arguments that were not supplied
-    if localedir is None:
-        localedir = _default_localedir
-    if languages is None:
-        languages = []
-        for envar in ('LANGUAGE', 'LC_ALL', 'LC_MESSAGES', 'LANG'):
-            val = os.environ.get(envar)
-            if val:
-                languages = val.split(':')
-                break
-        if 'C' not in languages:
-            languages.append('C')
-    # now normalize and expand the languages
-    nelangs = []
-    for lang in languages:
-        for nelang in _expand_lang(lang):
-            if nelang not in nelangs:
-                nelangs.append(nelang)
-    # select a language
-    if all:
-        result = []
-    else:
-        result = None
-    for lang in nelangs:
-        if lang == 'C':
-            break
-        mofile = os.path.join(localedir, lang, 'LC_MESSAGES', '%s.mo' % domain)
-        if os.path.exists(mofile):
-            if all:
-                result.append(mofile)
-            else:
-                return mofile
-    return result
-
-
-
-# a mapping between absolute .mo file path and Translation object
-_translations = {}
-
-def translation(domain, localedir=None, languages=None,
-                class_=None, fallback=False, codeset=None):
-    if class_ is None:
-        class_ = GNUTranslations
-    mofiles = find(domain, localedir, languages, all=1)
-    if not mofiles:
-        if fallback:
-            return NullTranslations()
-        raise IOError(ENOENT, 'No translation file found for domain', domain)
-    # Avoid opening, reading, and parsing the .mo file after it's been done
-    # once.
-    result = None
-    for mofile in mofiles:
-        key = (class_, os.path.abspath(mofile))
-        t = _translations.get(key)
-        if t is None:
-            with open(mofile, 'rb') as fp:
-                t = _translations.setdefault(key, class_(fp))
-        # Copy the translation object to allow setting fallbacks and
-        # output charset. All other instance data is shared with the
-        # cached object.
-        t = copy.copy(t)
-        if codeset:
-            t.set_output_charset(codeset)
-        if result is None:
-            result = t
-        else:
-            result.add_fallback(t)
-    return result
-
-
-def install(domain, localedir=None, unicode=False, codeset=None, names=None):
-    t = translation(domain, localedir, fallback=True, codeset=codeset)
-    t.install(unicode, names)
-
-
-
-# a mapping b/w domains and locale directories
-_localedirs = {}
-# a mapping b/w domains and codesets
-_localecodesets = {}
-# current global domain, `messages' used for compatibility w/ GNU gettext
-_current_domain = 'messages'
-
-
-def textdomain(domain=None):
-    global _current_domain
-    if domain is not None:
-        _current_domain = domain
-    return _current_domain
-
-
-def bindtextdomain(domain, localedir=None):
-    global _localedirs
-    if localedir is not None:
-        _localedirs[domain] = localedir
-    return _localedirs.get(domain, _default_localedir)
-
-
-def bind_textdomain_codeset(domain, codeset=None):
-    global _localecodesets
-    if codeset is not None:
-        _localecodesets[domain] = codeset
-    return _localecodesets.get(domain)
-
-
-def dgettext(domain, message):
-    try:
-        t = translation(domain, _localedirs.get(domain, None),
-                        codeset=_localecodesets.get(domain))
-    except IOError:
-        return message
-    return t.gettext(message)
-
-def ldgettext(domain, message):
-    try:
-        t = translation(domain, _localedirs.get(domain, None),
-                        codeset=_localecodesets.get(domain))
-    except IOError:
-        return message
-    return t.lgettext(message)
-
-def dngettext(domain, msgid1, msgid2, n):
-    try:
-        t = translation(domain, _localedirs.get(domain, None),
-                        codeset=_localecodesets.get(domain))
-    except IOError:
-        if n == 1:
-            return msgid1
-        else:
-            return msgid2
-    return t.ngettext(msgid1, msgid2, n)
-
-def ldngettext(domain, msgid1, msgid2, n):
-    try:
-        t = translation(domain, _localedirs.get(domain, None),
-                        codeset=_localecodesets.get(domain))
-    except IOError:
-        if n == 1:
-            return msgid1
-        else:
-            return msgid2
-    return t.lngettext(msgid1, msgid2, n)
-
-def gettext(message):
-    return dgettext(_current_domain, message)
-
-def lgettext(message):
-    return ldgettext(_current_domain, message)
-
-def ngettext(msgid1, msgid2, n):
-    return dngettext(_current_domain, msgid1, msgid2, n)
-
-def lngettext(msgid1, msgid2, n):
-    return ldngettext(_current_domain, msgid1, msgid2, n)
-
-# dcgettext() has been deemed unnecessary and is not implemented.
-
-# James Henstridge's Catalog constructor from GNOME gettext.  Documented usage
-# was:
-#
-#    import gettext
-#    cat = gettext.Catalog(PACKAGE, localedir=LOCALEDIR)
-#    _ = cat.gettext
-#    print _('Hello World')
-
-# The resulting catalog object currently don't support access through a
-# dictionary API, which was supported (but apparently unused) in GNOME
-# gettext.
-
-Catalog = translation

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Fri Mar 20 19:59:39 2015
From: jython-checkins at python.org (jim.baker)
Date: Fri, 20 Mar 2015 18:59:39 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Copy_over_CPython_version_o?=
	=?utf-8?q?f_gettext_module?=
Message-ID: <20150320185938.63588.7886@psf.io>

https://hg.python.org/jython/rev/6c0fe6e598ba
changeset:   7624:6c0fe6e598ba
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 20 12:59:27 2015 -0600
summary:
  Copy over CPython version of gettext module

files:
  CPythonLib.includes |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/CPythonLib.includes b/CPythonLib.includes
--- a/CPythonLib.includes
+++ b/CPythonLib.includes
@@ -74,6 +74,7 @@
 functools.py
 genericpath.py
 getopt.py
+gettext.py
 glob.py
 gopherlib.py
 hashlib.py

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 21 05:07:28 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 21 Mar 2015 04:07:28 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Update_bin/jython=2Epy_so_c?=
 =?utf-8?q?mdline-defined_Java_props_always_take_precedence?=
Message-ID: <20150321040727.11896.48923@psf.io>

https://hg.python.org/jython/rev/4ead3d557b37
changeset:   7625:4ead3d557b37
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 20 22:07:19 2015 -0600
summary:
  Update bin/jython.py so cmdline-defined Java props always take precedence

files:
  src/shell/jython.py |  20 ++++++++++++--------
  1 files changed, 12 insertions(+), 8 deletions(-)


diff --git a/src/shell/jython.py b/src/shell/jython.py
--- a/src/shell/jython.py
+++ b/src/shell/jython.py
@@ -208,7 +208,7 @@
         return os.path.join(self.jython_home, "javalib", "profile.jar")
 
     def set_encoding(self):
-        if "JAVA_ENCODING" not in os.environ and self.uname == "darwin":
+        if "JAVA_ENCODING" not in os.environ and self.uname == "darwin" and "file.encoding" not in self.args.properties:
             self.args.properties["file.encoding"] = "UTF-8"
 
     def convert(self, arg):
@@ -233,25 +233,29 @@
         args = [self.java_command]
         args.extend(self.java_opts)
         args.extend(self.args.java)
-        for k, v in self.args.properties.iteritems():
-            args.append("-D%s=%s" % (self.convert(k), self.convert(v)))
         if self.args.boot:
             args.append("-Xbootclasspath/a:%s" % self.convert_path(self.classpath))
             if self.args.classpath:
                 args.extend(["-classpath", self.convert_path(self.args.classpath)])
         else:
             args.extend(["-classpath", self.convert_path(self.classpath)])
-        args.append("-Dpython.home=%s" % self.convert_path(self.jython_home))
-        args.append("-Dpython.executable=%s" % self.convert_path(self.executable))
-        args.append("-Dpython.launcher.uname=%s" % self.uname)
+        if "python.home" not in self.args.properties:
+            args.append("-Dpython.home=%s" % self.convert_path(self.jython_home))
+        if "python.executable" not in self.args.properties:
+            args.append("-Dpython.executable=%s" % self.convert_path(self.executable))
+        if "python.launcher.uname" not in self.args.properties:
+            args.append("-Dpython.launcher.uname=%s" % self.uname)
         # determine if is-a-tty for the benefit of running on cygwin - mintty doesn't behave like
         # a standard windows tty and so JNR posix doesn't detect it properly
-        args.append("-Dpython.launcher.tty=%s" % str(os.isatty(sys.stdin.fileno())).lower())
-        if self.uname == "cygwin":
+        if "python.launcher.tty" not in self.args.properties:
+            args.append("-Dpython.launcher.tty=%s" % str(os.isatty(sys.stdin.fileno())).lower())
+        if self.uname == "cygwin" and "python.console" not in self.args.properties:
             args.append("-Dpython.console=org.python.core.PlainConsole")
         if self.args.profile:
             args.append("-XX:-UseSplitVerifier")
             args.append("-javaagent:%s" % self.convert_path(self.java_profile_agent))
+        for k, v in self.args.properties.iteritems():
+            args.append("-D%s=%s" % (self.convert(k), self.convert(v)))
         args.append("org.python.util.jython")
         if self.args.help:
             args.append("--help")

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 21 05:54:58 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 21 Mar 2015 04:54:58 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Ensure_that_compiled_=24py?=
 =?utf-8?q?=2Eclass_files_are_included_for_standalone=2C_full_jars?=
Message-ID: <20150321045458.30734.833@psf.io>

https://hg.python.org/jython/rev/72cf10235676
changeset:   7626:72cf10235676
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 20 22:54:54 2015 -0600
summary:
  Ensure that compiled $py.class files are included for standalone, full jars

Changed task dependencies to ensure compiled Python classes are built.

Fixed problem such that standalone jar (jython-standalone.jar) was
being built with jarjar task, but this was unnecessary and caused a
problem: the shading had already been applied in building
jython.jar. Switched to the standard jar task, which allowed for the
compiled $py.class files to be directly added, without jarjar task
moving them in the resulting jar.

Fixes http://bugs.jython.org/issue2282

files:
  build.xml |  14 +++++++-------
  1 files changed, 7 insertions(+), 7 deletions(-)


diff --git a/build.xml b/build.xml
--- a/build.xml
+++ b/build.xml
@@ -569,7 +569,7 @@
             includesfile="${jython.base.dir}/CoreExposed.includes"/>
     </target>
 
-    <target name="jar-complete" depends="jar">
+    <target name="jar-complete" depends="jar, pycompile">
         <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask" classpath="extlibs/jarjar-1.4.jar"/>
         <jarjar destfile="${dist.dir}/${jython.deploy.jar}">
             <zipfileset src="${dist.dir}/${jython.dev.jar}"/>
@@ -646,12 +646,12 @@
         </jarjar>
     </target>
 
-<target name="jar-standalone" depends="jar-complete, copy-lib">
+    <target name="jar-standalone" depends="jar-complete">
         <taskdef name="jarjar" classname="com.tonicsystems.jarjar.JarJarTask" classpath="extlibs/jarjar-1.4.jar"/>
-        <jarjar destfile="${dist.dir}/${jython.standalone.jar}">
-            <zipfileset src="${dist.dir}/${jython.deploy.jar}"/>
-            <fileset dir="${dist.dir}" includes="Lib/**"/>
-            <manifest>
+        <jar destfile="${dist.dir}/${jython.standalone.jar}">
+	  <zipfileset src="${dist.dir}/${jython.deploy.jar}"/>
+	  <fileset dir="${dist.dir}" includes="Lib/**" excludes="Lib/test/**" />
+          <manifest>
                 <attribute name="Main-Class" value="org.python.util.jython" />
                 <attribute name="Built-By" value="${user.name}" />
                 <!-- info section. ATTN: no blanks, no '.' in the names -->
@@ -665,7 +665,7 @@
                     <attribute name="debug" value="${debug}" />
                 </section>
             </manifest>
-        </jarjar>
+	</jar>
     </target>
 
     <target name="jar" depends="compile,expose">

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 21 06:51:42 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 21 Mar 2015 05:51:42 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Remove_obsolete_reference_t?=
 =?utf-8?q?o_jython=2Ebat_=28now_jython=2Eexe=29_in_build=2Exml?=
Message-ID: <20150321055142.16924.45873@psf.io>

https://hg.python.org/jython/rev/7f85568b378f
changeset:   7627:7f85568b378f
user:        Jim Baker <jim.baker at rackspace.com>
date:        Fri Mar 20 23:51:36 2015 -0600
summary:
  Remove obsolete reference to jython.bat (now jython.exe) in build.xml

This fixes the regrtest for Windows. Should have been changed as part
of the fix for http://bugs.jython.org/issue1491

files:
  build.xml |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/build.xml b/build.xml
--- a/build.xml
+++ b/build.xml
@@ -1034,7 +1034,7 @@
       </exec>
     </target>
     <target name="regrtest-windows" if="os.family.windows">
-      <exec executable="${dist.dir}/bin/jython.bat">
+      <exec executable="${dist.dir}/bin/jython.exe">
         <arg value="${dist.dir}/Lib/test/regrtest.py"/>
         <!-- Only run the tests that are expected to work on Jython -->
         <arg value="--expected"/>

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 21 07:21:35 2015
From: jython-checkins at python.org (jim.baker)
Date: Sat, 21 Mar 2015 06:21:35 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Rebuild_bin/jython=2Eexe_wi?=
	=?utf-8?q?th_PyInstaller?=
Message-ID: <20150321062135.7259.96302@psf.io>

https://hg.python.org/jython/rev/198b9d6e30d0
changeset:   7628:198b9d6e30d0
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sat Mar 21 00:21:30 2015 -0600
summary:
  Rebuild bin/jython.exe with PyInstaller

Underlying bin/python.py was updated in 7625:4ead3d557b37

files:
  src/shell/jython.exe |  Bin 
  1 files changed, 0 insertions(+), 0 deletions(-)


diff --git a/src/shell/jython.exe b/src/shell/jython.exe
index 0e68e65aded720b141b9b4e2daf0d0dca4d46dc9..6c355cfcca0b46834c948785d323e2d3d93a36cf
GIT binary patch
[stripped]

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 21 22:17:13 2015
From: jython-checkins at python.org (santoso.wijaya)
Date: Sat, 21 Mar 2015 21:17:13 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Introduce_and_use_a_non-buf?=
 =?utf-8?q?fered_readinto=28=29_method_for_BufferedReader=2E?=
Message-ID: <20150321211713.24254.1074@psf.io>

https://hg.python.org/jython/rev/b597f478d5ab
changeset:   7629:b597f478d5ab
user:        Santoso Wijaya <santoso.wijaya at gmail.com>
date:        Sat Mar 21 14:16:30 2015 -0700
summary:
  Introduce and use a non-buffered readinto() method for BufferedReader.

Introduce and use an overloaded non-buffered readinto() method for
BufferedReader when #read(int size) is given an explicit size to read
from its underlying stream.

This is to bring observable behavior to par with CPython in
issue #1011.

files:
  Lib/test/test_os_jy.py                     |  58 +++++++++-
  src/org/python/core/io/BufferedIOBase.java |  17 ++-
  src/org/python/core/io/BufferedReader.java |  28 +++-
  src/org/python/core/io/FileIO.java         |  16 ++-
  4 files changed, 104 insertions(+), 15 deletions(-)


diff --git a/Lib/test/test_os_jy.py b/Lib/test/test_os_jy.py
--- a/Lib/test/test_os_jy.py
+++ b/Lib/test/test_os_jy.py
@@ -4,13 +4,15 @@
 
 Made for Jython.
 """
+import os
+import sys
+import glob
 import array
 import errno
-import glob
-import os
+import struct
+import unittest
 import subprocess
-import sys
-import unittest
+
 from test import test_support
 from java.io import File
 
@@ -74,6 +76,54 @@
             content = f.read()
         self.assertEqual(content, 2 * b'onetwothree')
 
+    def test_issue1011(self):
+        # prepare the input file containing 256 bytes of sorted byte-sized numbers
+        fd = file(test_support.TESTFN, 'wb')
+        try:
+            for x in range(256):
+                fd.write(chr(x))
+        finally:
+            fd.close()
+
+        # reopen in read/append mode
+        fd = file(test_support.TESTFN, 'rb+')
+        try:
+            # read forward from the beginning
+            for x in range(256):
+                pos = fd.tell()
+                self.assertEqual(pos, x,
+                        '[forward] before read: pos should be %d but is %d' % (x, pos))
+
+                # read just one byte
+                c = struct.unpack('B', fd.read(1))[0]
+
+                pos = fd.tell()
+                self.assertEqual(pos, x + 1,
+                        '[forward] after read: pos should be %d but is %d' % (x + 1, pos))
+                
+                self.assertEqual(c, x)
+
+            # read backward from the end
+            fd.seek(-1, os.SEEK_END)
+            for x in range(255, -1, -1):
+                pos = fd.tell()
+                self.assertEqual(pos, x,
+                        '[backward] before read: pos should be %d but is %d' % (x, pos))
+
+                # read just one byte
+                c = ord(fd.read(1))
+
+                pos = fd.tell()
+                self.assertEqual(pos, x + 1,
+                        '[backward] after read: pos should be %d but is %d' % (x + 1, pos))
+                
+                self.assertEqual(c, x)
+
+                if x > 0:
+                    fd.seek(-2, os.SEEK_CUR)
+        finally:
+            fd.close()
+
 
 class OSDirTestCase(unittest.TestCase):
 
diff --git a/src/org/python/core/io/BufferedIOBase.java b/src/org/python/core/io/BufferedIOBase.java
--- a/src/org/python/core/io/BufferedIOBase.java
+++ b/src/org/python/core/io/BufferedIOBase.java
@@ -26,7 +26,7 @@
         }
 
         ByteBuffer bytes = ByteBuffer.allocate(size);
-        readinto(bytes);
+        readinto(bytes, false);
         // flip()ing here is more convenient as there's no real use
         // case for appending to buffers returned from read. readinto
         // doesn't/shouldn't flip()
@@ -58,6 +58,21 @@
     }
 
     /**
+     * Read up to bytes.remaining() bytes into the given ByteBuffer, but control
+     * whether to attempt to buffer the rest of the underlying stream.
+     *
+     * Returns the number of bytes read (0 for EOF)
+     *
+     * @param bytes a ByteBuffer to read bytes into
+     * @param buffered whether to buffer the remaining bytes of the stream, if we can
+     * @return the amount of data read as an int
+     */
+    protected int readinto(ByteBuffer bytes, boolean buffered) {
+        unsupported("readinto");
+        return -1;
+    }
+
+    /**
      * Write the given ByteBuffer to the IO stream.
      *
      * Returns the number of bytes written, which may be less than
diff --git a/src/org/python/core/io/BufferedReader.java b/src/org/python/core/io/BufferedReader.java
--- a/src/org/python/core/io/BufferedReader.java
+++ b/src/org/python/core/io/BufferedReader.java
@@ -29,6 +29,11 @@
 
     @Override
     public int readinto(ByteBuffer bytes) {
+        return readinto(bytes, true);
+    }
+
+    @Override
+    protected int readinto(ByteBuffer bytes, boolean buffered) {
         int size = bytes.remaining();
 
         if (size == 0) {
@@ -46,15 +51,26 @@
 
         // Drain the buffer then request more from the RawIO
         bytes.put(buffer);
-        buffer.clear();
 
-        // Only attempt one read. The buffering layer should not wait
-        // for more data (block) to fulfill the entire read
-        long read = rawIO.readinto(new ByteBuffer[] {bytes, buffer});
-        read -= buffer.flip().limit();
+        long read;
+        if (buffered) {
+            // Only attempt one read. The buffering layer should not wait
+            // for more data (block) to fulfill the entire read.
+            // Use scatter I/O read operation to read (potentially) farther
+            // than suggested in `bytes.remaining()` to fill in internal
+            // buffer `buffer`, that can be used to fulfill future read requests.
+            buffer.clear();
+            read = rawIO.readinto(new ByteBuffer[]{bytes, buffer});
+            read -= buffer.flip().limit();
+        } else {
+            // Clear internal buffer and only read up to specified size
+            // (implicit in `bytes.remaining()`) from the underlying stream
+            clear();
+            read = rawIO.readinto(bytes);
+        }
 
         // This is an int after subtracting the buffer size anyway
-        return (int)read;
+        return (int) read;
     }
 
     @Override
diff --git a/src/org/python/core/io/FileIO.java b/src/org/python/core/io/FileIO.java
--- a/src/org/python/core/io/FileIO.java
+++ b/src/org/python/core/io/FileIO.java
@@ -30,6 +30,14 @@
  */
 public class FileIO extends RawIOBase {
 
+    // would be nicer if we directly imported from os, but crazy to do so
+    // since in python code itself
+    private static class os {
+        public static final int SEEK_SET = 0;
+        public static final int SEEK_CUR = 1;
+        public static final int SEEK_END = 2;
+    }
+
     /** The underlying file channel */
     private FileChannel fileChannel;
 
@@ -204,7 +212,7 @@
      */
     private void initPosition() {
         if (appending) {
-            seek(0, 2);
+            seek(0, os.SEEK_END);
         } else if (writing && !reading) {
             try {
                 fileChannel.truncate(0);
@@ -367,12 +375,12 @@
         checkClosed();
         try {
             switch (whence) {
-            case 0:
+            case os.SEEK_SET:
                 break;
-            case 1:
+            case os.SEEK_CUR:
                 pos += fileChannel.position();
                 break;
-            case 2:
+            case os.SEEK_END:
                 pos += fileChannel.size();
                 break;
             default:

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sat Mar 21 22:47:14 2015
From: jython-checkins at python.org (santoso.wijaya)
Date: Sat, 21 Mar 2015 21:47:14 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Correction=3A_The_parent_ch?=
 =?utf-8?q?angeset_was_for_issue_=231793=2E?=
Message-ID: <20150321214714.4207.44929@psf.io>

https://hg.python.org/jython/rev/3548f6836c66
changeset:   7630:3548f6836c66
user:        Santoso Wijaya <santoso.wijaya at gmail.com>
date:        Sat Mar 21 14:46:35 2015 -0700
summary:
  Correction: The parent changeset was for issue #1793.

files:
  Lib/test/test_os_jy.py |  2 +-
  1 files changed, 1 insertions(+), 1 deletions(-)


diff --git a/Lib/test/test_os_jy.py b/Lib/test/test_os_jy.py
--- a/Lib/test/test_os_jy.py
+++ b/Lib/test/test_os_jy.py
@@ -76,7 +76,7 @@
             content = f.read()
         self.assertEqual(content, 2 * b'onetwothree')
 
-    def test_issue1011(self):
+    def test_issue1793(self):
         # prepare the input file containing 256 bytes of sorted byte-sized numbers
         fd = file(test_support.TESTFN, 'wb')
         try:

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sun Mar 22 05:27:20 2015
From: jython-checkins at python.org (frank.wierzbicki)
Date: Sun, 22 Mar 2015 04:27:20 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Prepare_for_2=2E7rc1=2E?=
Message-ID: <20150322042720.30712.79948@psf.io>

https://hg.python.org/jython/rev/d9660aa5cc8a
changeset:   7631:d9660aa5cc8a
user:        Frank Wierzbicki <fwierzbicki at gmail.com>
date:        Sun Mar 22 04:27:25 2015 +0000
summary:
  Prepare for 2.7rc1.

files:
  README.txt |  13 +++++--------
  build.xml  |   8 ++++----
  2 files changed, 9 insertions(+), 12 deletions(-)


diff --git a/README.txt b/README.txt
--- a/README.txt
+++ b/README.txt
@@ -1,16 +1,13 @@
-Welcome to Jython 2.7b4
+Welcome to Jython 2.7rc1
 =======================
 
-This is the fourth beta release of the 2.7 version of Jython. Thanks to
-Amobee (http://www.amobee.com/) for sponsoring this release.
-Thanks to all who contribute to Jython.
+This is the first release candidate of the 2.7 version of Jython. Thanks to
+Amobee (http://www.amobee.com/) for sponsoring this release.  Thanks to all who
+contribute to Jython.
 
-Jython 2.7b4 includes many bug fixes and code stabilization in prep for
+Jython 2.7rc1 includes many bug fixes and code stabilization in prep for
 release candidates.
 
-As a beta release we are concentrating on bug fixing and stabilizion for a
-production release.
-
 Please see the NEWS file for detailed release notes.
 
 The release was compiled on OSX with JDK 7 and requires JDK 7 to run.
diff --git a/build.xml b/build.xml
--- a/build.xml
+++ b/build.xml
@@ -84,15 +84,15 @@
         <property name="PY_RELEASE_LEVEL_SNAPSHOT" value="170"/> <!-- 0xAA -->
 
         <!-- The current version info -->
-        <property name="jython.version" value="2.7b4+"/>
-        <property name="jython.version.noplus" value="2.7b4"/>
+        <property name="jython.version" value="2.7rc1+"/>
+        <property name="jython.version.noplus" value="2.7rc1"/>
         <property name="jython.major_version" value="2"/>
         <property name="jython.minor_version" value="7"/>
         <property name="jython.micro_version" value="0"/>
-        <property name="jython.release_level" value="${PY_RELEASE_LEVEL_BETA}"/>
+        <property name="jython.release_level" value="${PY_RELEASE_LEVEL_GAMMA}"/>
         <!-- Usually zero, only used for alpha, beta and candidate versions
              where it must be greater than zero. -->
-        <property name="jython.release_serial" value="4"/>
+        <property name="jython.release_serial" value="1"/>
         <property name="jython.java.version" value="1.7"/>
 
         <condition property="do.snapshot.build">

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sun Mar 22 05:27:50 2015
From: jython-checkins at python.org (frank.wierzbicki)
Date: Sun, 22 Mar 2015 04:27:50 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_Added_tag_v2=2E7rc1_for_cha?=
	=?utf-8?q?ngeset_d9660aa5cc8a?=
Message-ID: <20150322042750.70030.3802@psf.io>

https://hg.python.org/jython/rev/3d8067c56a1d
changeset:   7632:3d8067c56a1d
user:        Frank Wierzbicki <fwierzbicki at gmail.com>
date:        Sun Mar 22 04:27:55 2015 +0000
summary:
  Added tag v2.7rc1 for changeset d9660aa5cc8a

files:
  .hgtags |  1 +
  1 files changed, 1 insertions(+), 0 deletions(-)


diff --git a/.hgtags b/.hgtags
--- a/.hgtags
+++ b/.hgtags
@@ -86,3 +86,4 @@
 e270e881c84cbe519035c215fde09677362a4953 v2.7b4
 e270e881c84cbe519035c215fde09677362a4953 v2.7b4
 17d18c6dde9693129d977affb8ca20406d6a6585 v2.7b4
+d9660aa5cc8af0ce8c839b9712301c180cc4803e v2.7rc1

-- 
Repository URL: https://hg.python.org/jython

From jython-checkins at python.org  Sun Mar 29 07:40:46 2015
From: jython-checkins at python.org (jim.baker)
Date: Sun, 29 Mar 2015 05:40:46 +0000
Subject: [Jython-checkins] =?utf-8?q?jython=3A_sys=2Eps1_and_sys=2Eps2_sho?=
 =?utf-8?q?uld_only_be_defined_if_interactive?=
Message-ID: <20150329054046.27946.24991@psf.io>

https://hg.python.org/jython/rev/23c3effa5d4f
changeset:   7633:23c3effa5d4f
user:        Jim Baker <jim.baker at rackspace.com>
date:        Sat Mar 28 23:40:41 2015 -0600
summary:
  sys.ps1 and sys.ps2 should only be defined if interactive

Raised on this thread -
http://sourceforge.net/p/jython/mailman/message/33651092/

files:
  Lib/test/test_sys_jy.py                     |  16 +++++++++-
  src/org/python/core/PySystemState.java      |   5 +-
  src/org/python/util/InteractiveConsole.java |  14 ++++++++
  3 files changed, 32 insertions(+), 3 deletions(-)


diff --git a/Lib/test/test_sys_jy.py b/Lib/test/test_sys_jy.py
--- a/Lib/test/test_sys_jy.py
+++ b/Lib/test/test_sys_jy.py
@@ -254,6 +254,19 @@
                 stdout=subprocess.PIPE)
             self.assertEqual(p.stdout.read().decode("utf-8"), zhongwen)
 
+class InteractivePromptTest(unittest.TestCase):
+    # TODO ps1, ps2 being defined for interactive usage should be
+    # captured by test_doctest, however, it would be ideal to add
+    # pexpect tests (using CPython).
+
+    def test_prompts_not_defined_if_not_interactive(self):
+        p = subprocess.Popen(
+            [sys.executable, '-c',
+             'import sys;' \
+             'print hasattr(sys, "ps1");' \
+             'print hasattr(sys, "ps2");'],
+            stdout=subprocess.PIPE)
+        self.assertEqual(''.join(p.stdout.read().split()), 'FalseFalse')
 
 def test_main():
     test_support.run_unittest(
@@ -262,7 +275,8 @@
         SyspathResourceTest,
         SyspathUnicodeTest,
         SysEncodingTest,
-        SysArgvTest
+        SysArgvTest,
+        InteractivePromptTest
     )
 
 if __name__ == "__main__":
diff --git a/src/org/python/core/PySystemState.java b/src/org/python/core/PySystemState.java
--- a/src/org/python/core/PySystemState.java
+++ b/src/org/python/core/PySystemState.java
@@ -147,8 +147,9 @@
     public PyList path_hooks;
     public PyObject path_importer_cache;
 
-    public PyObject ps1 = new PyString(">>> ");
-    public PyObject ps2 = new PyString("... ");
+    // Only defined if interactive, see https://docs.python.org/2/library/sys.html#sys.ps1
+    public PyObject ps1 = PyAttributeDeleted.INSTANCE;
+    public PyObject ps2 = PyAttributeDeleted.INSTANCE;
 
     public PyObject executable;
 
diff --git a/src/org/python/util/InteractiveConsole.java b/src/org/python/util/InteractiveConsole.java
--- a/src/org/python/util/InteractiveConsole.java
+++ b/src/org/python/util/InteractiveConsole.java
@@ -5,6 +5,7 @@
 import org.python.core.PyBuiltinFunctionSet;
 import org.python.core.PyException;
 import org.python.core.PyObject;
+import org.python.core.PyString;
 import org.python.core.PySystemState;
 import org.python.core.__builtin__;
 
@@ -111,6 +112,19 @@
      * @param file from which to read commands, or if <code>null</code>, read the console.
      */
     public void interact(String banner, PyObject file) {
+        PyObject old_ps1 = systemState.ps1;
+        PyObject old_ps2 = systemState.ps2;
+        systemState.ps1 = new PyString(">>> ");
+        systemState.ps2 = new PyString("... ");
+        try {
+            _interact(banner, file);
+        } finally {
+            systemState.ps1 = old_ps1;
+            systemState.ps2 = old_ps2;
+        }
+    }
+
+    public void _interact(String banner, PyObject file) {
         if (banner != null) {
             write(banner);
             write("\n");

-- 
Repository URL: https://hg.python.org/jython