[pypy-dev] translate.py failes
holger krekel
holger at merlinux.eu
Thu Sep 4 09:34:08 CEST 2008
Hi Vetoshkin,
i think for translate.py you rather need to run py.execnet over
CPython or through a translated pypy-c - both of which
should work fine.
I think you ran it through py.py which means starting
a PyPy interpreter on top of CPython. Then running
py.execnet.PopenGateway() would need to create another
process that also runs through py.py - would be very
slow - not sure it's worth trying to make it work.
So i suggest you continue your experiments with
plain CPython or the translated pypy-c.
best & thanks,
holger
On Thu, Sep 04, 2008 at 01:43 +0600, Vetoshkin Nikita wrote:
> Playing with py.execnet failes too. Full traceback attached.
>
> P.S. Seens like a hard start with my trials in PyPy =)
>
> Armin Rigo wrote:
>> Hi,
>>
>> On Thu, Aug 28, 2008 at 10:34:51PM +0600, Vetoshkin Nikita wrote:
>>> Is there a direction, where I could digg to help? Something to begin with.
>>> I don't have enough experience but I'd like to take part (even very
>>> small) in this project.
>>
>> Sure. Do you understand the issue? It's related to the way a Unix
>> process starts a new subprocess: os.fork() followed by os.execv() in the
>> child process. The issue is that os.fork() fails if the parent process'
>> total memory is more than half of your RAM+swap, because it tries to
>> create a copy of the process, and the copy would also need more than
>> half of your RAM+swap. (In reality, os.fork() doesn't copy the memory
>> at all, but creates "shared pages" of memory so that memory pages are
>> duplicated only when one of the two processes really modifies it; but
>> still os.fork() has to mark all the memory as *potentially* used, and
>> raises MemoryError if there isn't enough.)
>>
>> A solution might be to split translate.py in two processes: translate.py
>> would start a subprocess when it starts, and then do all the work
>> (consuming a lot of RAM); but when it needs to start new processes, e.g.
>> call gcc, it would instead ask the subprocess to start the new
>> processes. More precisely, the code in pypy.translator.tool.cbuild
>> would stop using distutils directly, and instead ask the subprocess to
>> use distutils. It looks even easy to do with the help of a
>> py.execnet.PopenGateway() created when translate.py starts (see the py
>> lib documentation at http://codespeak.net/py/).
>>
>>
>> A bientot,
>>
>> Armin
>
> --
> Nikita
> >>>> gw = py.execnet.PopenGateway()
> faking <type '_socket.socket'>
> faking <class 'socket.timeout'>
> Traceback (most recent call last):
> File "./pypy/bin/py.py", line 152, in <module>
> sys.exit(main_(sys.argv))
> File "./pypy/bin/py.py", line 138, in main_
> con.interact(banner)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line
> 133, in interact
> code.InteractiveConsole.interact(self, banner)
> File "/usr/lib64/python2.5/code.py", line 239, in interact
> more = self.push(line)
> File "/usr/lib64/python2.5/code.py", line 261, in push
> more = self.runsource(source, self.filename)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line
> 192, in runsource
> main.run_toplevel(self.space, doit, verbose=self.verbose)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/main.py", line 103, in
> run_toplevel
> f()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/interactive.py", line
> 184, in doit
> code.exec_code(self.space, self.w_globals, self.w_globals)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in
> exec_code
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in
> run
> return self.execute_frame()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117,
> in execute_frame
> executioncontext)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79,
> in dispatch
> next_instr = self.handle_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89,
> in handle_bytecode
> next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235,
> in dispatch_bytecode
> res = getattr(self, methodname)(oparg, next_instr)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 695,
> in LOAD_ATTR
> w_value = f.space.getattr(w_obj, w_attributename)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669,
> in getattr
> return DescrOperation.getattr(self, w_obj, w_name)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 134, in getattr
> return space._handle_getattribute(w_descr, w_obj, w_name)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 147, in _handle_getattribute
> return space.get_and_call_function(w_descr, w_obj, w_name)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 89, in get_and_call_function
> return descr.funccall(w_obj, *args_w)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 56,
> in funccall
> return code.fastcall_2(self.space, self, args_w[0], args_w[1])
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in
> fastcall_2
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in
> run
> return self.execute_frame()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117,
> in execute_frame
> executioncontext)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79,
> in dispatch
> next_instr = self.handle_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89,
> in handle_bytecode
> next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235,
> in dispatch_bytecode
> res = getattr(self, methodname)(oparg, next_instr)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881,
> in CALL_FUNCTION
> w_result = f.space.call_valuestack(w_function, nargs, f)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line
> 736, in call_valuestack
> return w_func.funccall_valuestack(nargs, frame)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81,
> in funccall_valuestack
> frame.peekvalue(0))
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in
> fastcall_2
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in
> run
> return self.execute_frame()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117,
> in execute_frame
> executioncontext)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79,
> in dispatch
> next_instr = self.handle_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89,
> in handle_bytecode
> next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235,
> in dispatch_bytecode
> res = getattr(self, methodname)(oparg, next_instr)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881,
> in CALL_FUNCTION
> w_result = f.space.call_valuestack(w_function, nargs, f)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line
> 736, in call_valuestack
> return w_func.funccall_valuestack(nargs, frame)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 81,
> in funccall_valuestack
> frame.peekvalue(0))
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pycode.py", line 191, in
> fastcall_2
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in
> run
> return self.execute_frame()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117,
> in execute_frame
> executioncontext)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79,
> in dispatch
> next_instr = self.handle_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89,
> in handle_bytecode
> next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235,
> in dispatch_bytecode
> res = getattr(self, methodname)(oparg, next_instr)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 881,
> in CALL_FUNCTION
> w_result = f.space.call_valuestack(w_function, nargs, f)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line
> 736, in call_valuestack
> return w_func.funccall_valuestack(nargs, frame)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 88,
> in funccall_valuestack
> frame.peekvalue(0))
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663,
> in fastcall_4
> w_result = self.fastfunc_4(space, w1, w2, w3, w4)
> File "</var/tmp/pypy-svn/dist/py/code/source.py:213>", line 3, in
> fastfunc_importhook_4
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 180, in importhook
> w_mod = absolute_import(space, modulename, 0, w_fromlist,
> tentative=0)
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 192, in absolute_import
> w_fromlist, tentative)
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 211, in _absolute_import
> tentative=tentative)
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 272, in load_part
> w(partname))
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 90, in try_import_mod
> magic, timestamp, stream.readall())
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 499, in load_compiled_module
> code_w.exec_code(space, w_dic, w_dic)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in
> exec_code
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in
> run
> return self.execute_frame()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117,
> in execute_frame
> executioncontext)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79,
> in dispatch
> next_instr = self.handle_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89,
> in handle_bytecode
> next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235,
> in dispatch_bytecode
> res = getattr(self, methodname)(oparg, next_instr)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 760,
> in IMPORT_NAME
> w_locals, w_fromlist)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/baseobjspace.py", line
> 712, in call_function
> return w_func.funccall(*args_w)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 62,
> in funccall
> args_w[1], args_w[2], args_w[3])
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 663,
> in fastcall_4
> w_result = self.fastfunc_4(space, w1, w2, w3, w4)
> File "</var/tmp/pypy-svn/dist/py/code/source.py:213>", line 3, in
> fastfunc_importhook_4
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 180, in importhook
> w_mod = absolute_import(space, modulename, 0, w_fromlist,
> tentative=0)
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 192, in absolute_import
> w_fromlist, tentative)
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 211, in _absolute_import
> tentative=tentative)
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 272, in load_part
> w(partname))
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 90, in try_import_mod
> magic, timestamp, stream.readall())
> File "/var/tmp/pypy-svn/dist/pypy/module/__builtin__/importing.py",
> line 499, in load_compiled_module
> code_w.exec_code(space, w_dic, w_dic)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 27, in
> exec_code
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 91, in
> run
> return self.execute_frame()
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyframe.py", line 117,
> in execute_frame
> executioncontext)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 79,
> in dispatch
> next_instr = self.handle_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 89,
> in handle_bytecode
> next_instr = self.dispatch_bytecode(co_code, next_instr, ec)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 235,
> in dispatch_bytecode
> res = getattr(self, methodname)(oparg, next_instr)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/pyopcode.py", line 769,
> in IMPORT_STAR
> import_all_from(f.space, w_module, w_locals)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 839,
> in appcaller
> return space.call_args(w_func, args)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 98, in call_args
> return w_obj.call_args(args)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37,
> in call_args
> return self.code.funcrun(self, args)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 483,
> in funcrun
> return BuiltinCode.funcrun_obj(self, func, None, args)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/gateway.py", line 491,
> in funcrun_obj
> w_result = activation._run(space, scope_w)
> File "</var/tmp/pypy-svn/dist/py/code/source.py:213>", line 3, in
> _run_UWS_ObjSpace_W_Root_W_Root
> File
> "/var/tmp/pypy-svn/dist/pypy/_cache/pyopcode_4922ecf55c7db5d81d0c4a63f92bdd44.py",
> line 59, in import_all_from
> w_2 = space.getattr(w_module, gs___dict__)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 669,
> in getattr
> return DescrOperation.getattr(self, w_obj, w_name)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 134, in getattr
> return space._handle_getattribute(w_descr, w_obj, w_name)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 140, in _handle_getattribute
> return space.get_and_call_function(w_descr, w_obj, w_name)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/descroperation.py", line
> 89, in get_and_call_function
> return descr.funccall(w_obj, *args_w)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 69,
> in funccall
> return self.call_args(Arguments(self.space, list(args_w)))
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/function.py", line 37,
> in call_args
> return self.code.funcrun(self, args)
> File "/var/tmp/pypy-svn/dist/pypy/interpreter/eval.py", line 61, in
> funcrun
> return frame.run()
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 171, in
> run
> return self.space.wrap(result)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 475,
> in wrap
> items_w = [(self.wrap(k), self.wrap(v)) for (k, v) in x.iteritems()]
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/objspace.py", line 522,
> in wrap
> return fake_object(self, x)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 20, in
> fake_object
> ft = fake_type(x)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 58, in
> fake_type
> faked_type = really_build_fake_type(cpy_type)
> File "/var/tmp/pypy-svn/dist/pypy/objspace/std/fake.py", line 108, in
> really_build_fake_type
> assert cpy_type.__base__ is basestring
> AssertionError
> _______________________________________________
> pypy-dev at codespeak.net
> http://codespeak.net/mailman/listinfo/pypy-dev
--
collaborative expert contracting: http://merlinux.eu
PyPy Python/Compiler tool chain: http://codespeak.net/pypy
pylib py.test/greenlets/svn APIs: http://pylib.org
More information about the Pypy-dev
mailing list