[ python-Bugs-1163563 ] Sub threads execute in restricted mode
SourceForge.net
noreply at sourceforge.net
Thu Jun 16 13:32:29 CEST 2005
Bugs item #1163563, was opened at 2005-03-15 09:56
Message generated for change (Comment added) made by yetanothermax
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1163563&group_id=5470
Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: Threads
Group: Python 2.4
Status: Open
Resolution: None
Priority: 8
Submitted By: anothermax (yetanothermax)
Assigned to: Nobody/Anonymous (nobody)
Summary: Sub threads execute in restricted mode
Initial Comment:
I'm using the JEP product which allows integration
of Java with Python (see http://jepp.sourceforge.net) via
starting a Python interpreter in the same process as the
JVM.
This integrates with python via the C interface, allowing
the user to 'eval' python code (amongst other features).
It seems that since Python 2.3.5 any threads (using the
threading module) created via code that has been
evaluated through the jep.eval() interface (i.e.Python C
interface )are executed in restricted mode and cannot,
for example, create files. Code that is just evaled (i.e not
in a subthread) thread has no restrictions.
The error reported is:
IOError: file() constructor not accessible in restricted
mode
(see http://sourceforge.net/forum/forum.php?
thread_id=1247793&forum_id=376782) - I've given a JEP
specific example here.
There seems to be a similar problem with mod_python
(see
http://www.modpython.org/pipermail/mod_python/2005-
January/017129.html)
Reading through the release notes for Python 2.3.5
I see:
Bug #754449: threading.Thread will no longer mask
exceptions raised during interpreter shutdown with
another exception caused by attempting to output the
initial exception. This fix also includes a backport of rev.
1.41 from HEAD.
This might be the problem as it seems to involve the
porting of 2.4 threading code back to the 2.3 tree.
I've attached a Java file which shows the problem when
using JEP.
The error output is:
Exception in thread Thread-1:
Traceback (most recent call last):
File "C:\Python24\Lib\threading.py", line 442, in
__bootstrap
File "<string>", line 5, in run
IOError: file() constructor not accessible in restricted
mode
2.4.1c1 (#63, Mar 10 2005, 10:36:41) [MSC v.1310 32
bit (Intel)]
Creating file from main thread...
Done
Creating file from sub thread...
Done
----------------------------------------------------------------------
>Comment By: anothermax (yetanothermax)
Date: 2005-06-16 13:32
Message:
Logged In: YES
user_id=1218811
Have you tried the threadtest.c with a debug build after
applying the patch to threadmodule.c?
The patch is here
http://sourceforge.net/tracker/index.php?
func=detail&aid=1203393&group_id=5470&atid=305470
Cheers,
Max
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2005-06-16 11:58
Message:
Logged In: YES
user_id=6656
Hmm. threadtest.c just crashed for me with a debug build:
Fatal Python error: Invalid thread state for this thread
Abort trap
This is in the call to Py_NewInterpreter() ... (eek).
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-05-17 10:49
Message:
Logged In: YES
user_id=1218811
I've created a patch for threadmodule.c
http://sourceforge.net/tracker/index.php?
func=detail&aid=1203393&group_id=5470&atid=305470
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-05-12 10:48
Message:
Logged In: YES
user_id=1218811
It looks like with Python versions > 2.3.4 threads are off limits
for multiple interpreters.
If that's your definition of screwed then I guess the answer's
yes.
I'm surprised more mod_python users haven't been bitten by
this.
Shall I raise theproblem on the list (I've already done so on
comp.lang.python a while ago)
----------------------------------------------------------------------
Comment By: Michael Hudson (mwh)
Date: 2005-05-12 09:43
Message:
Logged In: YES
user_id=6656
I saw the subject and thought "Hmm, should raise the priority of this so it
gets at least thought about hard by the next release", then saw that you
did that...
> How do we get this solved for the next Python release?
I don't know. It's possible it should be raised on python-dev (it seems
fairly quiet just now, so it might be a good time :).
Basically, it seems that the whole multiple interpreters facility is screwed,
correct?
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-05-12 09:35
Message:
Logged In: YES
user_id=1218811
This problem prevents us from upgrading to Python 2.4.
I've attached a another 'fixed' threadmodule.c which is based
upon the lastest CVS release.(Only change from the last
threadmodule.c attachment is the removal of a '\n' from an
error message).
The threadtest.c program demonstrates the problem (seems
to be the use of Py_NewInterpreter() with the
PyGIL_StateXXX functions in the current threadmodule.c).
How do we get this solved for the next Python release?
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-04-21 14:05
Message:
Logged In: YES
user_id=1218811
I've attached a file (threadtest.c) that is adapted from the
the way JEP uses Py_NewInterpreter.
This runs correctly
under 2.3.4 but fails under 2.3.5 and later Pythons.
----------------------------------------------------------------------
Comment By: mrjohnson (mrjohnson0)
Date: 2005-04-21 07:51
Message:
Logged In: YES
user_id=1044885
In interactive mode, Jep uses Py_NewInterpreter,
Py_CompileString, and PyRun_String for its eval().
This file handles java -> python:
http://204.156.146.230/cgi-bin/viewcvs.cgi/jep/pyembed.c?rev=1.29&content-type=text/vnd.viewcvs-markup
You'd be looking for pyembed_eval().
----------------------------------------------------------------------
Comment By: Mark Hammond (mhammond)
Date: 2005-04-21 05:11
Message:
Logged In: YES
user_id=14198
Can anyone tell me specifically what Python C API function
is involved here? ie, the "entry point" that these external
threads use to call into Python.
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-04-20 13:51
Message:
Logged In: YES
user_id=1218811
I didn't see jbelmote's comment, perhaps my updated
threadmodule.c will reopen the #1010677 bug.
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-04-20 13:39
Message:
Logged In: YES
user_id=1218811
Here's the updated threadmodule.c for 2.4.1
----------------------------------------------------------------------
Comment By: anothermax (yetanothermax)
Date: 2005-04-20 13:36
Message:
Logged In: YES
user_id=1218811
I've finally got access to a C compiler and it looks like
the 'pure' Python changes for #754449 are in the clear
(apologies to bcannon).
The problem actually seems to relate to changes to
threadmodule.c of the Python Modules directory.
In versions greater than 2.3.4 it uses the PyGILState_XXX
constructs t_bootstrap() function.
These do not seem to be compatible with running sub
interpreters as JEP (and probably mod_python) do.
The python documentation
(http://docs.python.org/api/threads.html) says "Note that the
PyGILState_*() functions assume there is only one global
interpreter (created automatically by Py_Initialize()). Python
still supports the creation of additional interpreters (using
Py_NewInterpreter()), but mixing multiple interpreters and the
PyGILState_*() API is unsupported. "
I've compiled a new versions of the python2X.dll for both 2.3.5
and 2.4.1 changing the use of these PyGIL_StateXXX
constructs back to the 2.3.4 form and this seems to resolve
the problem (I've attached the updated threadmodule.c files
for both 2.3.5 and 2.4.1).
----------------------------------------------------------------------
Comment By: John Belmonte (jbelmonte)
Date: 2005-04-16 22:27
Message:
Logged In: YES
user_id=282299
This problem also occurs on Linux. Note however that
I've traced the source of the symptom to a different bug
fix, namely [1010677] thread Module Breaks
PyGILState_Ensure()
(http://sourceforge.net/tracker/index.php?func=detail&aid=1010677&group_id=5470&atid=305470).
Specifically, the change from v2.56 to v2.56.8.1 of
threadmodule.c.
----------------------------------------------------------------------
Comment By: Brett Cannon (bcannon)
Date: 2005-04-16 21:21
Message:
Logged In: YES
user_id=357491
To answer the comment on bug #754449 and the email I got
from someone, there is no way my fix for that bug would
cause this. I only touched the 'threading' module, nothing
else. That is in pure Python and thus has no direct way of
modifying what does and does not execute in restricted mode.
If I understand how restricted execution works that is all
set at the C level.
Plus the fix only deals with the teardown code; it in no way
interfaces with how threads are executed. All changed code
come after the thread and completed execution. The only
thing it does ahead of time is store a reference to stdout.
In other words it ain't my fault to the best of my
knowledge. =)
----------------------------------------------------------------------
Comment By: Alan Davies (goatpunch)
Date: 2005-03-31 08:24
Message:
Logged In: YES
user_id=967140
The same problem definately does occur with mod_python,
I've found it to occur with Python 2.3.5 and 2.4 on Win32:
http://www.modpython.org/pipermail/mod_python/2005-
March/017802.html
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1163563&group_id=5470
More information about the Python-bugs-list
mailing list