From cfbolz at gmx.de Tue Jul 1 00:58:51 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 01 Jul 2008 00:58:51 +0200 Subject: [pypy-dev] channel logs Message-ID: <4869652B.5050801@gmx.de> Hi all, just wanted to note that the braintone logs seem to work at the moment, so you can read the as a stopgap measure here: http://merlinux.de/braintone/chat.freenode.net/pypy/last/50/ The format is not so nice, so I would actually really like pybot back :-). Cheers, Carl Friedrich From marius at pov.lt Tue Jul 1 11:26:01 2008 From: marius at pov.lt (Marius Gedminas) Date: Tue, 1 Jul 2008 12:26:01 +0300 Subject: [pypy-dev] channel logs In-Reply-To: <4869652B.5050801@gmx.de> References: <4869652B.5050801@gmx.de> Message-ID: <20080701092601.GA27912@fridge.pov.lt> On Tue, Jul 01, 2008 at 12:58:51AM +0200, Carl Friedrich Bolz wrote: > Hi all, > > just wanted to note that the braintone logs seem to work at the moment, > so you can read the as a stopgap measure here: > > http://merlinux.de/braintone/chat.freenode.net/pypy/last/50/ > > The format is not so nice, so I would actually really like pybot back :-). Assuming you're on FreeNode, I could easily set up IRC logs that look like this: http://mg.pov.lt/maemo-irclog/latest.log.html Marius Gedminas -- Thus spake the master programmer: "After three days without programming, life becomes meaningless." -- Geoffrey James, "The Tao of Programming" -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: Digital signature URL: From tismer at stackless.com Tue Jul 1 12:16:35 2008 From: tismer at stackless.com (Christian Tismer) Date: Tue, 01 Jul 2008 12:16:35 +0200 Subject: [pypy-dev] channel logs In-Reply-To: <20080701092601.GA27912@fridge.pov.lt> References: <4869652B.5050801@gmx.de> <20080701092601.GA27912@fridge.pov.lt> Message-ID: <486A0403.6040804@stackless.com> Marius Gedminas wrote: > On Tue, Jul 01, 2008 at 12:58:51AM +0200, Carl Friedrich Bolz wrote: >> Hi all, >> >> just wanted to note that the braintone logs seem to work at the moment, >> so you can read the as a stopgap measure here: >> >> http://merlinux.de/braintone/chat.freenode.net/pypy/last/50/ >> >> The format is not so nice, so I would actually really like pybot back :-). > > Assuming you're on FreeNode, I could easily set up IRC logs that look > like this: http://mg.pov.lt/maemo-irclog/latest.log.html My pybot is working, again. It stopped working, when I did a dist-upgrade of my debian system. It runs fine in foreground, but when I try to deamonize it, it pretends to work, but doesn't. Will try to solve that, or switch to another bot. Meanwhile pybot is running foreground in a screen session. ciao - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From arigo at tunes.org Tue Jul 1 18:39:35 2008 From: arigo at tunes.org (Armin Rigo) Date: Tue, 1 Jul 2008 18:39:35 +0200 Subject: [pypy-dev] Small dicts Message-ID: <20080701163935.GA7076@code0.codespeak.net> Hi all, "withsmalldicts" is not enabled by default, I suppose because we measured that it didn't help. However I've found a bug which I think means that even when the option is enabled, they won't be used in most common cases. The bug is that the option is checked in EmptyDictImplementation.setitem() but not in EmptyDictImplementation.setitem_str(): the latter always creates a non-small StrDictImplementation. Anyone feels like fixing and re-measuring this? Armin From fijall at gmail.com Fri Jul 4 01:29:26 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Jul 2008 01:29:26 +0200 Subject: [pypy-dev] [pypy-svn] r56275 - pypy/extradoc/talk/ep2008 In-Reply-To: <20080703203052.A66CB169F57@codespeak.net> References: <20080703203052.A66CB169F57@codespeak.net> Message-ID: <693bc9ab0807031629sbdea4a6p8f1fd9146b210157@mail.gmail.com> On Thu, Jul 3, 2008 at 10:30 PM, wrote: > Author: hpk > Date: Thu Jul 3 22:30:52 2008 > New Revision: 56275 > > Modified: > pypy/extradoc/talk/ep2008/status.txt > Log: > review, cutting out some negative not too telling > statements and adding a bit to plans from my perspective > I don't like this changes completely. I would like to say clearly what's not in our plans. From fijall at gmail.com Fri Jul 4 06:53:29 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 4 Jul 2008 06:53:29 +0200 Subject: [pypy-dev] problems running pypy-jvm Message-ID: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> First of all, I cannot get jar to work. It's complaining about '@' in front of paths. Even if I remove it, it still cannot find Main class. When I run it by hand I get: debug: WARNING: library path not found, using compiled-in sys.path 'import site' failed Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' What's wrong? Cheers, fijal From holger at merlinux.de Fri Jul 4 08:05:59 2008 From: holger at merlinux.de (holger krekel) Date: Fri, 4 Jul 2008 08:05:59 +0200 Subject: [pypy-dev] [pypy-svn] r56275 - pypy/extradoc/talk/ep2008 In-Reply-To: <693bc9ab0807031629sbdea4a6p8f1fd9146b210157@mail.gmail.com> References: <20080703203052.A66CB169F57@codespeak.net> <693bc9ab0807031629sbdea4a6p8f1fd9146b210157@mail.gmail.com> Message-ID: <20080704060559.GL759@trillke.net> On Fri, Jul 04, 2008 at 01:29 +0200, Maciej Fijalkowski wrote: > On Thu, Jul 3, 2008 at 10:30 PM, wrote: > > Author: hpk > > Date: Thu Jul 3 22:30:52 2008 > > New Revision: 56275 > > > > Modified: > > pypy/extradoc/talk/ep2008/status.txt > > Log: > > review, cutting out some negative not too telling > > statements and adding a bit to plans from my perspective > > > > I don't like this changes completely. I would like to say clearly > what's not in our plans. maybe mention it explicitely in the "cleanup sprint" slide before? making the very last slide about what we are not going to do is not a good ending IMO. holger From anto.cuni at gmail.com Fri Jul 4 09:57:33 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Fri, 04 Jul 2008 09:57:33 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> Message-ID: <486DD7ED.9090107@gmail.com> Hi Maciek, Maciej Fijalkowski wrote: > First of all, I cannot get jar to work. It's complaining about '@' in > front of paths. Even if I remove it, it still cannot find Main class. what do you exactly mean? Could you paste the error message please? > When I run it by hand I get: > > debug: WARNING: library path not found, using compiled-in sys.path > 'import site' failed > Error calling sys.excepthook: > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: cannot import name 'curdir' > > What's wrong? No clue. I've translated pypy-jvm yesterday just after the merging of the less-meta-instance branch, and it works fine: antocuni at viper goal $ ./pypy-jvm Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``nothing is true'' >>>> If I move pypy-jvm to another dir and rename my wc to prevent it to find the compiled-in sys.path, I get another error, which seems reasonable: antocuni at viper tmp $ ./pypy-jvm debug: WARNING: library path not found, using compiled-in sys.path 'import site' failed Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 Type "help", "copyright", "credits" or "license" for more information. debug: OperationError: debug: operror-type: ImportError debug: operror-value: No module named _pypy_interact Which revision is your pypy-jvm? Any clue why it doesn't find the library path? What ./pypy-jvm -c 'import sys; print sys.path' prints? ciao, Anto From victor.stinner at haypocalc.com Mon Jul 7 00:23:38 2008 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 7 Jul 2008 00:23:38 +0200 Subject: [pypy-dev] Play with fuzzing Message-ID: <200807070023.38555.victor.stinner@haypocalc.com> Hi, I wrote a fuzzing "framework" called Fusil and this week I wrote a fuzzer for Python. The idea is quite simple: for a module, - list all functions, classes and class methods - call a function with random arguments (of random types) - instanciate a class with random arguments - if the class is created correctly, call methods with random arguments Example: --------------------- 8< ----------------------------------- print "Call 39/40: linuxaudiodev.open()" try: linuxaudiodev.open( # argument 1/2 u"\u62C0\uFBD7\uB46A\u55E0\uFB7B\uD392\u7CEE", # argument 2/2 52.682, ) except Exception, err: print >>stderr, "ERROR: %s" % err --------------------- 8< ----------------------------------- I tried it on CPython 2.5 and then on CPython trunk (future 2.6). I found some bugs: http://bugs.python.org/issue3304 -> invalid call to PyMem_Free() in fileio_init() http://bugs.python.org/issue3306 -> audioop.findmax() crashs with negative length http://bugs.python.org/issue3303 -> invalid ref count on locale.strcoll() error http://bugs.python.org/issue3302 -> segfault on gettext(None) http://bugs.python.org/issue3301 -> DoS when lo is negative in bisect.insort_right() http://bugs.python.org/issue3299 -> invalid object destruction in re.finditer() Most bugs crash with a segmentation fault, abort or a denial of service. If you would like to try my fuzzer, use: (1) svn co http://fusil.hachoir.org/svn/trunk fusil (2) cd fusil (3) ./run_fusil.sh -p projects/python.py --fast --remove ALL The option --fast goes faster, --remove does remove session directory even if Python generated some files, and "ALL" test all modules. FUSIL IS NOT SAFE! So run it under a different using to avoid dangerous call to os.unlink(). Module list is hardcoded: it's the list of CPython modules written in C. More informations about Fusil: http://fusil.hachoir.org/trac -- Victor Stinner aka haypo http://www.haypocalc.com/blog/ From brunogola at gmail.com Mon Jul 7 16:31:36 2008 From: brunogola at gmail.com (Bruno Gola) Date: Mon, 7 Jul 2008 11:31:36 -0300 Subject: [pypy-dev] Changing the default magic number Message-ID: Hi, I'm writing code to support the new 2.5 generator stuff (PEP 342) and while working on that I found a code that verifies the magic number (in pyframe.py). The default magic in PyPy is the Python2.4 one, chatting with Carl we thought it's probably a good time to change it to the 2.5 value, as we are supporting Python 2.5. What do you think? -- Bruno Fialho Marques Gola http://www.brunogola.com.br Cel: (11) 9294-5883 From arigo at tunes.org Tue Jul 8 17:28:50 2008 From: arigo at tunes.org (Armin Rigo) Date: Tue, 8 Jul 2008 17:28:50 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: References: Message-ID: <20080708152850.GA8093@code0.codespeak.net> Hi Bruno, On Mon, Jul 07, 2008 at 11:31:36AM -0300, Bruno Gola wrote: > I'm writing code to support the new 2.5 generator stuff (PEP 342) and > while working on that I found a code that verifies the magic number > (in pyframe.py). The default magic in PyPy is the Python2.4 one, > chatting with Carl we thought it's probably a good time to change it > to the 2.5 value, as we are supporting Python 2.5. The important thing here is the opcodes that changed their semantics between 2.4 and 2.5. There is the YIELD opcode which is probably the one that you're referring to; in 2.5 it pushes a result on the value stack, but in 2.4 it didn't. There is also one of the IMPORT opcodes, which pops one more argument from the stack in 2.5. So the magic version number in PyPy should be changed to 2.5's at the same time as these two opcodes change their semantics in PyPy's compiler and in pyopcode.py. A bientot, Armin From cfbolz at gmx.de Wed Jul 9 14:30:24 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 09 Jul 2008 14:30:24 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: <20080708152850.GA8093@code0.codespeak.net> References: <20080708152850.GA8093@code0.codespeak.net> Message-ID: <4874AF60.1060807@gmx.de> Hi Armin, Armin Rigo wrote: > On Mon, Jul 07, 2008 at 11:31:36AM -0300, Bruno Gola wrote: >> I'm writing code to support the new 2.5 generator stuff (PEP 342) and >> while working on that I found a code that verifies the magic number >> (in pyframe.py). The default magic in PyPy is the Python2.4 one, >> chatting with Carl we thought it's probably a good time to change it >> to the 2.5 value, as we are supporting Python 2.5. > > The important thing here is the opcodes that changed their semantics > between 2.4 and 2.5. There is the YIELD opcode which is probably the > one that you're referring to; in 2.5 it pushes a result on the value > stack, but in 2.4 it didn't. There is also one of the IMPORT opcodes, > which pops one more argument from the stack in 2.5. So the magic > version number in PyPy should be changed to 2.5's at the same time as > these two opcodes change their semantics in PyPy's compiler and in > pyopcode.py. I had the impression that we use a PyPy-specific magic number anyway, in which case it doesn't matter that we change the magic only when all the new opcode semantics are fixed. Is that wrong? Have fun at the sprint! Carl Friedrich From fijall at gmail.com Thu Jul 10 09:24:34 2008 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 10 Jul 2008 09:24:34 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: <4874AF60.1060807@gmx.de> References: <20080708152850.GA8093@code0.codespeak.net> <4874AF60.1060807@gmx.de> Message-ID: <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> Magic in pyc files and magic as a number for pyopcode are different things IMO. On Wed, Jul 9, 2008 at 2:30 PM, Carl Friedrich Bolz wrote: > Hi Armin, > > Armin Rigo wrote: >> On Mon, Jul 07, 2008 at 11:31:36AM -0300, Bruno Gola wrote: >>> I'm writing code to support the new 2.5 generator stuff (PEP 342) and >>> while working on that I found a code that verifies the magic number >>> (in pyframe.py). The default magic in PyPy is the Python2.4 one, >>> chatting with Carl we thought it's probably a good time to change it >>> to the 2.5 value, as we are supporting Python 2.5. >> >> The important thing here is the opcodes that changed their semantics >> between 2.4 and 2.5. There is the YIELD opcode which is probably the >> one that you're referring to; in 2.5 it pushes a result on the value >> stack, but in 2.4 it didn't. There is also one of the IMPORT opcodes, >> which pops one more argument from the stack in 2.5. So the magic >> version number in PyPy should be changed to 2.5's at the same time as >> these two opcodes change their semantics in PyPy's compiler and in >> pyopcode.py. > > I had the impression that we use a PyPy-specific magic number anyway, in > which case it doesn't matter that we change the magic only when all the > new opcode semantics are fixed. Is that wrong? > > Have fun at the sprint! > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From brunogola at gmail.com Thu Jul 10 16:15:36 2008 From: brunogola at gmail.com (Bruno Gola) Date: Thu, 10 Jul 2008 11:15:36 -0300 Subject: [pypy-dev] Changing the default magic number In-Reply-To: <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> References: <20080708152850.GA8093@code0.codespeak.net> <4874AF60.1060807@gmx.de> <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> Message-ID: Hi, On Thu, Jul 10, 2008 at 4:24 AM, Maciej Fijalkowski wrote: > Magic in pyc files and magic as a number for pyopcode are different things IMO. Yes, in PyPy they are different things. The magic number in .pyc files compiled by PyPy is 1024. PyPy doesn't try to use .pyc generated by CPython and CPython will not try to use .pyc generated by PyPy (because of these magic number). But the magic I'm talking about is the default value for PyCode magic attribute. In pycode.py there is comment saying "value for Python 2.4.1" and a default value of 62061 for these attribute. These number is checked in many cases (or at least two) where, as Armin said, changes to the semantics of bytecodes have been made. It seems to me that these default value is always used, I could not find any place where a PyCode instance is created changing these value, but I may be wrong. Anyway, the semantics for YIELD_VALUE have already been changed (in 2.5-features branch) and i think it's good idea to change the other opcodes that have changed as well. The magic attribute is checked against the magic value for Python2.5 (and other) some times (probably when the opcode semantics changed and PyPy wanted to support both semantics) but as the default value is never changed (as far as I know) I think those checks are not being useful. I think we could change these value to the 2.5 one. I think it will not affect anything (only these opcodes that we are changing as well). Cheers, -- Bruno Fialho Marques Gola http://www.brunogola.com.br Cel: (11) 9294-5883 From zeroth at oddco.ca Thu Jul 10 21:57:06 2008 From: zeroth at oddco.ca (Tyler Laing) Date: Thu, 10 Jul 2008 12:57:06 -0700 Subject: [pypy-dev] Using pypy's translator? Message-ID: <3618a6e10807101257p68629f49k983a92c6643b8771@mail.gmail.com> Hello all. My name is Tyler, or Zeroth as my pseudonym. I'm currently undertaking a personal project, that seeks to explore pseudo-strict type checking for Python. More simply, it would analyze the code, and determine what methods/functions/operators each variable would need to support in the code, and then enforcing that all variables passed around follow the informal contract. It would merely enforce that the methods are present, and return the right(by right, meaning variables with the expected methods) types. It is an extension of duck typing; "If it walks like a duck, and talks like a duck, lets make sure they're expecting a duck!" I have been following the pypy project for a little while, and there are a tonne of very cool features. Of particular interest is the lexer/parser for Python. First, is it okay if I use it for my project(open source itself), and second, any advice on using it? Tutorials? Gotchas, warnings? Thanks in advance, and have lots of fun. -Tyler -- Visit my blog at http://oddco.ca/zeroth/zblog -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.de Sun Jul 13 18:00:37 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 13 Jul 2008 18:00:37 +0200 Subject: [pypy-dev] alias for --allopts? Message-ID: <20080713160037.GA759@trillke.net> Hi pypy-dev, would be good to rename the --faassen option sometime soon Martijn asked us multiple times already for it. What about "--cheetah" or "--falcon"? other suggestions? best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From cfbolz at gmx.de Sun Jul 13 18:14:04 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Sun, 13 Jul 2008 18:14:04 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713160037.GA759@trillke.net> References: <20080713160037.GA759@trillke.net> Message-ID: <487A29CC.7090602@gmx.de> holger krekel wrote: > would be good to rename the --faassen option sometime soon > Martijn asked us multiple times already for it. What about > "--cheetah" or "--falcon"? other suggestions? I don't really like --falcon, falcons can only be that fast when flying downwards (crashing?). How about --sailfish (fastest fish, only very slightly slower than a Cheeta) or maybe even --blackmamba (fastest snake)? Cheers, Carl Friedrich From eric at vanrietpaap.nl Sun Jul 13 18:43:58 2008 From: eric at vanrietpaap.nl (Eric van Riet Paap) Date: Sun, 13 Jul 2008 18:43:58 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <487A29CC.7090602@gmx.de> References: <20080713160037.GA759@trillke.net> <487A29CC.7090602@gmx.de> Message-ID: What about --fastest :-) cheers Eric On Sun, Jul 13, 2008 at 6:14 PM, Carl Friedrich Bolz wrote: > holger krekel wrote: > > would be good to rename the --faassen option sometime soon > > Martijn asked us multiple times already for it. What about > > "--cheetah" or "--falcon"? other suggestions? > > I don't really like --falcon, falcons can only be that fast when flying > downwards (crashing?). How about --sailfish (fastest fish, only very > slightly slower than a Cheeta) or maybe even --blackmamba (fastest snake)? > > Cheers, > > Carl Friedrich > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From holger at merlinux.de Sun Jul 13 18:52:47 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 13 Jul 2008 18:52:47 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <487A29CC.7090602@gmx.de> References: <20080713160037.GA759@trillke.net> <487A29CC.7090602@gmx.de> Message-ID: <20080713165247.GC759@trillke.net> On Sun, Jul 13, 2008 at 18:14 +0200, Carl Friedrich Bolz wrote: > holger krekel wrote: > > would be good to rename the --faassen option sometime soon > > Martijn asked us multiple times already for it. What about > > "--cheetah" or "--falcon"? other suggestions? > > I don't really like --falcon, falcons can only be that fast when flying > downwards (crashing?). How about --sailfish (fastest fish, only very > slightly slower than a Cheeta) or maybe even --blackmamba (fastest snake)? let's keep "blackmamba" as the release name for when we include the JIT by default :) --sailfish sounds slow to me (because "sailing" is not the fastest movement method i guess) but after reading wikipedia i kind of like it. i also like "--faastest", btw :) holger From arigo at tunes.org Sun Jul 13 23:19:01 2008 From: arigo at tunes.org (Armin Rigo) Date: Sun, 13 Jul 2008 23:19:01 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713160037.GA759@trillke.net> References: <20080713160037.GA759@trillke.net> Message-ID: <20080713211900.GA31170@code0.codespeak.net> Hi Holger, On Sun, Jul 13, 2008 at 06:00:37PM +0200, holger krekel wrote: > would be good to rename the --faassen option sometime soon > Martijn asked us multiple times already for it. What about > "--cheetah" or "--falcon"? other suggestions? "--allopts" is already an alias for it, and I changed the docs to mention it instead of --faassen. I also came up with some suggestion for the optimization options. What about introducing the --opt translation option (applying to all translations, not just targetpypystandalone) with the following possible values, which roughly correspond to the scheme used by gcc: --opt=0: all backendopt and other optimizations off; gc=boehm --opt=1: default backendopts, low inlining threadhold; gc=boehm --opt=size: same as opt=1 + remove_asserts --opt=mem: same as opt=size + gc=marknsweep (or some future low-ram gc) --opt=2: default backendopts; smallfuncset=5; gc=hybrid --opt=3: same as opt=2 + remove_asserts During a PyPy translation, we would control which pypy-specific options to use with the same scheme: a --pypyopt option (whose value is by default equal to the --opt option in order to reduce surprizes) according to a table like the following one: pypyopt option =================================== 2 3 ("objspace.opcodes.CALL_LIKELY_BUILTIN", True), 2 3 ("objspace.opcodes.CALL_METHOD", True), 3 ("translation.profopt", "..."), 2 3 ("objspace.std.withmultidict", True), 2 3 ("objspace.std.withshadowtracking", True), mem ("objspace.std.withsmallint", True), mem 2 3 ("objspace.std.withrangelist", True), 2 3 ("objspace.std.withmethodcache", True), mem 2 3 ("objspace.std.withprebuiltchar", True), 2 3 ("objspace.std.builtinshortcut", True), 2 3 ("objspace.std.optimized_list_getitem", True), 2 3 ("objspace.std.getattributeshortcut", True), mem ("objspace.std.withsharingdict", True), So this would generalize the --faassen and --allopts options. The fastest results would be obtained with --opt=3, --opt=size would try to minimize the executable size, --opt=mem would try to minimize run-time RAM usage. A bientot, Armin. From holger at merlinux.de Sun Jul 13 23:41:18 2008 From: holger at merlinux.de (holger krekel) Date: Sun, 13 Jul 2008 23:41:18 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713211900.GA31170@code0.codespeak.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> Message-ID: <20080713214118.GE759@trillke.net> Hi Armin, On Sun, Jul 13, 2008 at 23:19 +0200, Armin Rigo wrote: > On Sun, Jul 13, 2008 at 06:00:37PM +0200, holger krekel wrote: > > would be good to rename the --faassen option sometime soon > > Martijn asked us multiple times already for it. What about > > "--cheetah" or "--falcon"? other suggestions? > > "--allopts" is already an alias for it, and I changed the docs > to mention it instead of --faassen. ah, good. i only checked that "--allopts" is already there for a long time. > I also came up with some suggestion for the optimization options. > What about introducing the --opt translation option (applying to all > translations, not just targetpypystandalone) with the following possible > values, which roughly correspond to the scheme used by gcc: > > --opt=0: all backendopt and other optimizations off; gc=boehm > --opt=1: default backendopts, low inlining threadhold; gc=boehm > --opt=size: same as opt=1 + remove_asserts > --opt=mem: same as opt=size + gc=marknsweep (or some future low-ram gc) > --opt=2: default backendopts; smallfuncset=5; gc=hybrid > --opt=3: same as opt=2 + remove_asserts i like the idea. But I find mixing in the choice of GC a bit odd. what about defaulting to gc=hybrid and getting rid of the boehm dependency? > During a PyPy translation, we would control which pypy-specific options > to use with the same scheme: a --pypyopt option (whose value is by > default equal to the --opt option in order to reduce surprizes) > according to a table like the following one: > > pypyopt option > =================================== > > 2 3 ("objspace.opcodes.CALL_LIKELY_BUILTIN", True), > 2 3 ("objspace.opcodes.CALL_METHOD", True), > 3 ("translation.profopt", "..."), > 2 3 ("objspace.std.withmultidict", True), > 2 3 ("objspace.std.withshadowtracking", True), > mem ("objspace.std.withsmallint", True), > mem 2 3 ("objspace.std.withrangelist", True), > 2 3 ("objspace.std.withmethodcache", True), > mem 2 3 ("objspace.std.withprebuiltchar", True), > 2 3 ("objspace.std.builtinshortcut", True), > 2 3 ("objspace.std.optimized_list_getitem", True), > 2 3 ("objspace.std.getattributeshortcut", True), > mem ("objspace.std.withsharingdict", True), > > So this would generalize the --faassen and --allopts options. The > fastest results would be obtained with --opt=3, --opt=size would try to > minimize the executable size, --opt=mem would try to minimize run-time > RAM usage. Looks fine to me! cheers, holger From micahel at gmail.com Mon Jul 14 00:12:09 2008 From: micahel at gmail.com (Michael Hudson) Date: Mon, 14 Jul 2008 10:12:09 +1200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713214118.GE759@trillke.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> <20080713214118.GE759@trillke.net> Message-ID: 2008/7/14 holger krekel : > Hi Armin, > > On Sun, Jul 13, 2008 at 23:19 +0200, Armin Rigo wrote: >> On Sun, Jul 13, 2008 at 06:00:37PM +0200, holger krekel wrote: >> > would be good to rename the --faassen option sometime soon >> > Martijn asked us multiple times already for it. What about >> > "--cheetah" or "--falcon"? other suggestions? >> >> "--allopts" is already an alias for it, and I changed the docs >> to mention it instead of --faassen. > > ah, good. i only checked that "--allopts" is already there > for a long time. FWIW, I think I added --allopts when I was writing up one of the EU reports, not quite being able to bring myself to write about --faassen in serious prose... Cheers, mwh From holger at merlinux.de Mon Jul 14 16:05:18 2008 From: holger at merlinux.de (holger krekel) Date: Mon, 14 Jul 2008 16:05:18 +0200 Subject: [pypy-dev] EP sprint report / blog Message-ID: <20080714140518.GE22122@trillke.net> Hi pypy-dev, not sure how many pypy-dev subscribers are also reading the Blog regularly - we have been posting there about some EuroPythion 2008 events and also the sprint report, see http://morepypy.blogspot.com/ Btw, I'd find it nice to also have a blog->pypy-dev gateway - ideally one that includes relaying e-mail / comments to and fro. I wonder which blogging software might support something like this. cheers, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From cfbolz at gmx.de Mon Jul 14 16:30:33 2008 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 14 Jul 2008 16:30:33 +0200 Subject: [pypy-dev] EP sprint report / blog In-Reply-To: <20080714140518.GE22122@trillke.net> References: <20080714140518.GE22122@trillke.net> Message-ID: <487B6309.7010100@gmx.de> Hi Holger, holger krekel wrote: > not sure how many pypy-dev subscribers are also reading the > Blog regularly - we have been posting there about some > EuroPythion 2008 events and also the sprint report, see > http://morepypy.blogspot.com/ > > Btw, I'd find it nice to also have a blog->pypy-dev > gateway - ideally one that includes relaying > e-mail / comments to and fro. I wonder which > blogging software might support something like this. Sending e-mails to pypy-dev with the blog text is easy. I can set that up, if everybody thinks this is sensible. Also relaying the comments wouldn't be too hard. I have no clue whether somebody wrote something to push e-mail answers on the mailing list back as blog comments into the blog. Cheers, Carl Friedrich From holger at merlinux.de Mon Jul 14 16:44:16 2008 From: holger at merlinux.de (holger krekel) Date: Mon, 14 Jul 2008 16:44:16 +0200 Subject: [pypy-dev] EP sprint report / blog In-Reply-To: <487B6309.7010100@gmx.de> References: <20080714140518.GE22122@trillke.net> <487B6309.7010100@gmx.de> Message-ID: <20080714144416.GG22122@trillke.net> On Mon, Jul 14, 2008 at 16:30 +0200, Carl Friedrich Bolz wrote: > holger krekel wrote: > > not sure how many pypy-dev subscribers are also reading the > > Blog regularly - we have been posting there about some > > EuroPythion 2008 events and also the sprint report, see > > http://morepypy.blogspot.com/ > > > > Btw, I'd find it nice to also have a blog->pypy-dev > > gateway - ideally one that includes relaying > > e-mail / comments to and fro. I wonder which > > blogging software might support something like this. > > Sending e-mails to pypy-dev with the blog text is easy. I can set that > up, if everybody thinks this is sensible. Also relaying the comments > wouldn't be too hard. I have no clue whether somebody wrote something to > push e-mail answers on the mailing list back as blog comments into the blog. For relaying the comments things need to get properly threaded in mail readers and email->blog needs to work as well but quick Googling revealed nothing to me :( i find this strange with all the blogs and mailing lists out there. Without this integration, i don't know if it makes sense. but maybe someone knows some more or has other opinions. holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From arigo at tunes.org Mon Jul 14 17:27:28 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Jul 2008 17:27:28 +0200 Subject: [pypy-dev] Changing the default magic number In-Reply-To: References: <20080708152850.GA8093@code0.codespeak.net> <4874AF60.1060807@gmx.de> <693bc9ab0807100024n498f199bo67be7f096a950f9e@mail.gmail.com> Message-ID: <20080714152728.GA29978@code0.codespeak.net> Hi, On Thu, Jul 10, 2008 at 11:15:36AM -0300, Bruno Gola wrote: > I think we could change these value to the 2.5 one. I think it will > not affect anything (only these opcodes that we are changing as well). I believe that we are still trying to get code objects compiled by the underlying Python to run in PyPy. This is a case where the magic number in the PyCode can be different: it's used to know if we need to emulate the 2.4 or 2.5 behavior, depending on the underlying Python's version number. It would be a bit of a mess to remove this hack, though. For example the flow object space looks at the regular Python function object's func_code, turned into PyCode, so it must be able to accept PyCode objects that come from the underlying Python's compiler... A bientot, Armin. From arigo at tunes.org Mon Jul 14 17:29:37 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Jul 2008 17:29:37 +0200 Subject: [pypy-dev] Using pypy's translator? In-Reply-To: <3618a6e10807101257p68629f49k983a92c6643b8771@mail.gmail.com> References: <3618a6e10807101257p68629f49k983a92c6643b8771@mail.gmail.com> Message-ID: <20080714152937.GB29978@code0.codespeak.net> Hi, On Thu, Jul 10, 2008 at 12:57:06PM -0700, Tyler Laing wrote: > I have been following the pypy project for a little while, and there are a > tonne of very cool features. Of particular interest is the lexer/parser for > Python. First, is it okay if I use it for my project(open source itself), > and second, any advice on using it? Tutorials? Gotchas, warnings? Feel free to reuse any code. A warning is that our current lexer/parser is not too clean, though. A bientot, Armin. From arigo at tunes.org Mon Jul 14 17:32:06 2008 From: arigo at tunes.org (Armin Rigo) Date: Mon, 14 Jul 2008 17:32:06 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080713214118.GE759@trillke.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> <20080713214118.GE759@trillke.net> Message-ID: <20080714153206.GC29978@code0.codespeak.net> Hi Holger, On Sun, Jul 13, 2008 at 11:41:18PM +0200, holger krekel wrote: > i like the idea. But I find mixing in the choice of GC a bit odd. > what about defaulting to gc=hybrid and getting rid of > the boehm dependency? There are several reasons for wanting Boehm, but the main one is that translation is much faster and takes less memory, which is the whole point of an option like "--opt=0". The default value for --opt should probably be 2, so that by default we would get a good and Boehm-independent result. A bientot, Armin. From holger at merlinux.de Mon Jul 14 17:46:04 2008 From: holger at merlinux.de (holger krekel) Date: Mon, 14 Jul 2008 17:46:04 +0200 Subject: [pypy-dev] alias for --allopts? In-Reply-To: <20080714153206.GC29978@code0.codespeak.net> References: <20080713160037.GA759@trillke.net> <20080713211900.GA31170@code0.codespeak.net> <20080713214118.GE759@trillke.net> <20080714153206.GC29978@code0.codespeak.net> Message-ID: <20080714154604.GH22122@trillke.net> Hi Armin, On Mon, Jul 14, 2008 at 17:32 +0200, Armin Rigo wrote: > On Sun, Jul 13, 2008 at 11:41:18PM +0200, holger krekel wrote: > > i like the idea. But I find mixing in the choice of GC a bit odd. > > what about defaulting to gc=hybrid and getting rid of > > the boehm dependency? > > There are several reasons for wanting Boehm, but the main one is that > translation is much faster and takes less memory, which is the whole > point of an option like "--opt=0". > > The default value for --opt should probably be 2, so that by default we > would get a good and Boehm-independent result. sounds fine to me. i guess that we'd just drop "--faassen" then. holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From paskma at gmail.com Mon Jul 14 22:30:26 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Mon, 14 Jul 2008 22:30:26 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <486DD7ED.9090107@gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> Message-ID: <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> Hi pypy hackers :-), I have the same problem. pypy-jvm is unable to start in interactive mode: paskma at paskma:goal$ ./pypy-jvm 'import site' failed Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``choose your hack'' Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' Here is an investigation of sys.path, maybe it helps you: paskma at paskma:goal$ pwd /home/data/opt/svnpypy/dist/pypy/translator/goal paskma at paskma:goal$ ./pypy-c -c 'import sys; print sys.path' ['/home/data/opt/svnpypy/dist/pypy/translator/goal', '/home/data/opt/svnpypy/dist/pypy/lib', '/home/data/opt/svnpypy/dist/lib-python/modified-2.4.1', '/home/data/opt/svnpypy/dist/lib-python/2.4.1', '/usr/lib/python2.4/site-packages'] paskma at paskma:goal$ ./pypy-jvm -c 'import sys; print sys.path' 'import site' failed ['', '/home/data/opt/svnpypy/dist/pypy/lib', '/home/data/opt/svnpypy/dist/lib-python/modified-2.4.1', '/home/data/opt/svnpypy/dist/lib-python/2.4.1'] The sys.path of the pypy-jvm lacks '/home/data/opt/svnpypy/dist/pypy/translator/goal' and mainly '/usr/lib/python2.4/site-packages'. The translation of the pypy-jvm was done without any significant switches. Marek Pa?ka On Fri, Jul 4, 2008 at 09:57, Antonio Cuni wrote: > Hi Maciek, > > Maciej Fijalkowski wrote: >> First of all, I cannot get jar to work. It's complaining about '@' in >> front of paths. Even if I remove it, it still cannot find Main class. > > what do you exactly mean? Could you paste the error message please? > >> When I run it by hand I get: >> >> debug: WARNING: library path not found, using compiled-in sys.path >> 'import site' failed >> Error calling sys.excepthook: >> debug: OperationError: >> debug: operror-type: ImportError >> debug: operror-value: cannot import name 'curdir' >> >> What's wrong? > > No clue. I've translated pypy-jvm yesterday just after the merging of > the less-meta-instance branch, and it works fine: > > antocuni at viper goal $ ./pypy-jvm > Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``nothing is true'' > >>>> > > If I move pypy-jvm to another dir and rename my wc to prevent it to find > the compiled-in sys.path, I get another error, which seems reasonable: > > antocuni at viper tmp $ ./pypy-jvm > debug: WARNING: library path not found, using compiled-in sys.path > 'import site' failed > Python 2.4.1 (pypy 1.0.0 build 56259) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: No module named _pypy_interact > > Which revision is your pypy-jvm? Any clue why it doesn't find the > library path? > What ./pypy-jvm -c 'import sys; print sys.path' prints? > > ciao, > Anto > _______________________________________________ > pypy-dev at codespeak.net > http://codespeak.net/mailman/listinfo/pypy-dev > From anto.cuni at gmail.com Wed Jul 16 10:25:00 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 16 Jul 2008 10:25:00 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> Message-ID: <487DB05C.6040702@gmail.com> Marek Pa?ka wrote: > Hi pypy hackers :-), > > I have the same problem. pypy-jvm is unable to start in interactive mode: > > paskma at paskma:goal$ ./pypy-jvm > 'import site' failed > Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``choose your hack'' > Error calling sys.excepthook: > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: cannot import name 'curdir' that's very weird, I've never managed to reproduce this error. Could you put pypy-jvm.jar somewhere on the web so that I can try it on my machine, please? Or you could try to set DEBUG=True in translator/goal/app_main.py, recompile and see if we get more informative messages. thank you very much! ciao, Anto From paskma at gmail.com Wed Jul 16 11:08:42 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Wed, 16 Jul 2008 11:08:42 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <487DB05C.6040702@gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> Message-ID: <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> Hi, you can obtain it from here: http://147.228.64.164/downloads/pypy-jvm.jar And I am building DEBUG=True version right now :-) Marek On Wed, Jul 16, 2008 at 10:25, Antonio Cuni wrote: > Marek Pa?ka wrote: >> >> Hi pypy hackers :-), >> >> I have the same problem. pypy-jvm is unable to start in interactive mode: >> >> paskma at paskma:goal$ ./pypy-jvm >> 'import site' failed >> Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> ``choose your hack'' >> Error calling sys.excepthook: >> debug: OperationError: >> debug: operror-type: ImportError >> debug: operror-value: cannot import name 'curdir' > > that's very weird, I've never managed to reproduce this error. > Could you put pypy-jvm.jar somewhere on the web so that I can try it on my > machine, please? > > Or you could try to set DEBUG=True in translator/goal/app_main.py, recompile > and see if we get more informative messages. > > thank you very much! > ciao, > Anto > From paskma at gmail.com Wed Jul 16 11:46:49 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Wed, 16 Jul 2008 11:46:49 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> Message-ID: <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> You can download DEBUG=True version from here: http://147.228.64.164/downloads/pypy-jvm.jar-debugOn The error message is now little more specific: paskma at paskma:goal$ ./pypy-jvm 'import site' failed Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 Type "help", "copyright", "credits" or "license" for more information. ``peephope optimizations are what a Sufficiently Smart Compiler uses'' debug: exception-type: ImportError debug: exception-value: cannot import name 'curdir' debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' Marek On Wed, Jul 16, 2008 at 11:08, Marek Pa?ka wrote: > Hi, > > you can obtain it from here: > > http://147.228.64.164/downloads/pypy-jvm.jar > > And I am building DEBUG=True version right now :-) > > Marek > > On Wed, Jul 16, 2008 at 10:25, Antonio Cuni wrote: >> Marek Pa?ka wrote: >>> >>> Hi pypy hackers :-), >>> >>> I have the same problem. pypy-jvm is unable to start in interactive mode: >>> >>> paskma at paskma:goal$ ./pypy-jvm >>> 'import site' failed >>> Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 >>> Type "help", "copyright", "credits" or "license" for more information. >>> ``choose your hack'' >>> Error calling sys.excepthook: >>> debug: OperationError: >>> debug: operror-type: ImportError >>> debug: operror-value: cannot import name 'curdir' >> >> that's very weird, I've never managed to reproduce this error. >> Could you put pypy-jvm.jar somewhere on the web so that I can try it on my >> machine, please? >> >> Or you could try to set DEBUG=True in translator/goal/app_main.py, recompile >> and see if we get more informative messages. >> >> thank you very much! >> ciao, >> Anto >> > From anto.cuni at gmail.com Wed Jul 16 12:43:03 2008 From: anto.cuni at gmail.com (Antonio Cuni) Date: Wed, 16 Jul 2008 12:43:03 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> Message-ID: <487DD0B7.9030708@gmail.com> Marek Pa?ka wrote: > You can download DEBUG=True version from here: > > http://147.228.64.164/downloads/pypy-jvm.jar-debugOn it works fine on my machine. > The error message is now little more specific: > > paskma at paskma:goal$ ./pypy-jvm > 'import site' failed > Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 > Type "help", "copyright", "credits" or "license" for more information. > ``peephope optimizations are what a Sufficiently Smart Compiler uses'' > debug: exception-type: ImportError > debug: exception-value: cannot import name 'curdir' > debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 > Error calling sys.excepthook: > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: cannot import name 'curdir' weird. It fails when it tries to import curdir from os.path; could you try the following please? ./pypy-jvm -c 'import os.path; print os.path' ./pypy-jvm -c' import os.path; print dir(os.path)' or maybe just come on #pypy on freenode, so we can try real time debugging over irc :-). My nick is antocuni. ciao, Anto From paskma at gmail.com Wed Jul 16 13:38:38 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Wed, 16 Jul 2008 13:38:38 +0200 Subject: [pypy-dev] problems running pypy-jvm In-Reply-To: <487DD0B7.9030708@gmail.com> References: <693bc9ab0807032153w9af52a5x70ce69eabd0e11c@mail.gmail.com> <486DD7ED.9090107@gmail.com> <4ddb710e0807141330n2d3a90adv6292102bb0c3bdfa@mail.gmail.com> <487DB05C.6040702@gmail.com> <4ddb710e0807160208u4ba6dedcvbbb4f10b2eb92659@mail.gmail.com> <4ddb710e0807160246of167284pa59ae479fc2f4ef9@mail.gmail.com> <487DD0B7.9030708@gmail.com> Message-ID: <4ddb710e0807160438s64a7f818g5cd1ddf9a42db966@mail.gmail.com> The trace is always the same; the os module ties to import 'curdir'. I logged to #pypy. paskma at paskma:goal$ ./pypy-jvm -c 'import os.path; print os.path' 'import site' failed debug: exception-type: ImportError debug: exception-value: cannot import name 'curdir' debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' paskma at paskma:goal$ ./pypy-jvm -c 'import os.path; print dir(os.path)' 'import site' failed debug: exception-type: ImportError debug: exception-value: cannot import name 'curdir' debug: exception-tb: /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 Error calling sys.excepthook: debug: OperationError: debug: operror-type: ImportError debug: operror-value: cannot import name 'curdir' Marek On Wed, Jul 16, 2008 at 12:43, Antonio Cuni wrote: > Marek Pa?ka wrote: >> >> You can download DEBUG=True version from here: >> >> http://147.228.64.164/downloads/pypy-jvm.jar-debugOn > > it works fine on my machine. > >> The error message is now little more specific: >> >> paskma at paskma:goal$ ./pypy-jvm >> 'import site' failed >> Python 2.4.1 (pypy 1.0.0 build 56547) on linux2 >> Type "help", "copyright", "credits" or "license" for more information. >> ``peephope optimizations are what a Sufficiently Smart Compiler uses'' >> debug: exception-type: ImportError >> debug: exception-value: cannot import name 'curdir' >> debug: exception-tb: >> /home/data/opt/svnpypy/dist/lib-python/2.4.1/os.py:133 >> Error calling sys.excepthook: >> debug: OperationError: >> debug: operror-type: ImportError >> debug: operror-value: cannot import name 'curdir' > > weird. It fails when it tries to import curdir from os.path; could you try > the following please? > > ./pypy-jvm -c 'import os.path; print os.path' > ./pypy-jvm -c' import os.path; print dir(os.path)' > > > or maybe just come on #pypy on freenode, so we can try real time debugging > over irc :-). My nick is antocuni. > > ciao, > Anto > From ranjithkannikara at gmail.com Thu Jul 17 09:08:31 2008 From: ranjithkannikara at gmail.com (ranjith kannikara) Date: Thu, 17 Jul 2008 12:38:31 +0530 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. Message-ID: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> I have taken the gsoc 08 project of porting zope2 to python2.5. Through my way to the successful completetion of the project I have to implement Restricted python in Zope2. Chris Withers, consultant of simplistix have volunteered to give guidance and help in the RPython implementation I was also suggested by him to engage with the PyPy guys on their mailing list. and work with their guidance to re-implement RestrictedPython in a way that doesn't use AST hacks. Being a student I am not much familiar with the RPython yet need some help to get the RPython implementaion started.. How can I get started with the RPython implementation.. From holger at merlinux.eu Thu Jul 17 09:25:08 2008 From: holger at merlinux.eu (holger krekel) Date: Thu, 17 Jul 2008 09:25:08 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> Message-ID: <20080717072508.GF16827@trillke.net> Hello Ranjith, names are confusing here. PyPy's "RPython" is something else than Zope2's Restricted PYthon. I guess Chris Withers was refering to the two lightning talks that he and then Armin/Maciej gave at EuroPython 2008 both carrying "Restricted Python" in their title. For the future, I suggest to avoid the "R" term and rather talk about _sandboxing_ Python code which is what both talks/concepts were about. Indeed, PyPy provides a robust way of sandboxing its Python Interpreter, see http://codespeak.net/pypy/dist/pypy/doc/sandbox.html I am not sure how well this fits in your "porting zope2 to python2.5" project - but to me it looks like a good idea to go for using pypy-sandbox in the future instead of trying to get Zope2's "RestrictedPython" secure and ported to 2.5. cheers, holger On Thu, Jul 17, 2008 at 12:38 +0530, ranjith kannikara wrote: > I have taken the gsoc 08 project of porting zope2 to python2.5. > Through my way to the successful completetion of the project I have to > implement Restricted python in Zope2. Chris Withers, consultant of > simplistix have volunteered to give guidance and help in the RPython > implementation I was also suggested by him to engage with the PyPy > guys on their mailing list. and work with their guidance to > re-implement RestrictedPython in a way that doesn't use AST hacks. > Being a student I am not much familiar with the RPython yet need some > help to get the RPython implementaion started.. > > How can I get started with the RPython implementation.. -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From holger at merlinux.eu Thu Jul 17 11:40:09 2008 From: holger at merlinux.eu (holger krekel) Date: Thu, 17 Jul 2008 11:40:09 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <487EF825.2020704@simplistix.co.uk> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> Message-ID: <20080717094009.GA22440@trillke.net> Hey Chris, all, On Thu, Jul 17, 2008 at 08:43 +0100, Chris Withers wrote: > holger krekel wrote: >> names are confusing here. > > Indeed, I think Ranjith can be forgiven for getting confused given the > proliferation of similar names in this area ;-) > > For clarity, shall we agree to refer to things as: > > RestrictedPython - the current Zope 2 implementation found at > http://pypi.python.org/pypi/RestrictedPython > > Sandboxing - the technique of getting a secure and restricted python > environment running. > > RPython - some special in the pypy world ;-) "some special", aha :) i just added a FAQ entry: http://codespeak.net/pypy/dist/pypy/doc/faq.html#does-rpython-have-anything-to-do-with-zope-s-restricted-python >> Indeed, PyPy provides a robust way of sandboxing its Python >> Interpreter, see >> >> http://codespeak.net/pypy/dist/pypy/doc/sandbox.html > > Indeed, although as it is, I'm not sure how good a fit it is. > >> I am not sure how well this fits in your "porting zope2 to >> python2.5" project - but to me it looks like a good idea to go for >> using pypy-sandbox in the future instead of trying to get Zope2's >> "RestrictedPython" secure and ported to 2.5. > > Well, the rough feature request list is: > > - something that behaves like a python function > - can be pickled this is basically about storing user-provided scripts in the zodb, right? > - prevents people stripping security proxies from objects that are > passed in to the function hum, pypy-sandbox currently virtualizes at a low level (access to the Operating Systems files, io, etc.) and provides a *very* trustable way of running untrusted code. For providing higher and python level virtualization more work is needed so it might indeed not directly fit some use cases that RestrictedPython tries to satisfy. I am interested in this as are others, i think. Could you or someone else point to some concrete usages and use cases of the current RestrictedPython? > - prevents people importing things they shouldn't > - prevents people from getting hold of un-security-proxies-objects > - executes as fast as possible > - each function plays in its own sandbox and can be called many many > times, often simultaneously in different threads > - limits the amount of memory and processor they can hog (nice to have) the rest, i think, is quite doable, particularly limiting RAM/CPU resources. > I'm sure others will pitch in if I've forgotten anything... best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From chris at simplistix.co.uk Fri Jul 18 17:28:11 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 18 Jul 2008 16:28:11 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20080717094009.GA22440@trillke.net> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> Message-ID: <4880B68B.3080803@simplistix.co.uk> holger krekel wrote: >> - something that behaves like a python function >> - can be pickled > > this is basically about storing user-provided scripts > in the zodb, right? ...or relational database, or, etc ;-) >> - prevents people stripping security proxies from objects that are >> passed in to the function > > hum, pypy-sandbox currently virtualizes at a low level > (access to the Operating Systems files, io, etc.) and > provides a *very* trustable way of running untrusted code. Right, which is exactly what we want ;-) The security proxying is handled by a different method entirely. (The proxy objects are c extensions and prevent accessing attributes based on a pluggable set of rules) However, with access to unrestricted imports, all you'd need to do is import the unwrap method from the proxy wrapping module and you'd be fre of the jail ;-) > For providing higher and python level virtualization more work > is needed so it might indeed not directly fit some use cases > that RestrictedPython tries to satisfy. I am interested in > this as are others, i think. Could you or someone else point to > some concrete usages and use cases of the current RestrictedPython? - Download and install Zope 2.11.1. - Add a "Script (Python)" using the Zope management interface. - Write some python code or look at the default python code that starts off in the python script. The use case is basically threefold: - general distrust of code authored through a web browser (what if someone managed to break in somehow and insert source code for a PythonSript (the things Zope calls "Script (Python)" in its management interface), etc) - provide an environment where objects that shouldn't be accessible by a particular user are filtered by the environment and where objects can also prevent their attributes from being written to or read. (the user here doesn't write code, they are running other code... as a programmer, rather than having to write security checks all the time yourself, the environment does it for you) - allow customers to write scripts where they do so in an environment where they can't access things that they shouldn't be able to, be it data, or functions like os.system, etc. There's also a "nice to have" rather than a requirement of providing an environment where code can be safely run without it accidentally consuming all memory or processing resources on a machine. >> - prevents people importing things they shouldn't >> - prevents people from getting hold of un-security-proxies-objects >> - executes as fast as possible >> - each function plays in its own sandbox and can be called many many >> times, often simultaneously in different threads >> - limits the amount of memory and processor they can hog (nice to have) > > the rest, i think, is quite doable, particularly > limiting RAM/CPU resources. cool :-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From jacobidiego at gmail.com Sat Jul 19 05:00:06 2008 From: jacobidiego at gmail.com (Diego Jacobi) Date: Sat, 19 Jul 2008 00:00:06 -0300 Subject: [pypy-dev] I have a problem interpreting pypy functionality. Message-ID: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> Hi. I am trying to decifer what exactly is the result that i got from using pypy. As little as i understand, it can translate code from python to C. I have followed the "Getting started" guide, but it makes it too long to see the pypy features that i am confused. First at all, i have had some issues from trying to compile pypy into pypy-c, it claims for some error with cc, but cc is on my system. And i am using debian lenny, which is still under development. Im not sure if that can cause the problem. So, i couldnt see the code generated or the optimiced program running. I have found pypy, by looking for a translator from python to C. But what i was looking for is a translator which gives me a code without a python dependency. Maybe using stdlibs, or glib, or something, and libffi, so the python wrappers are converted too. In some of the tests in the guide i see that after compiling with t.compile_c() i can directly use the function in python. So, it is a python-module compiler? Also the guide doesnt mention where is saved the resultant module. I just want to translate a simple algoritm which i am using to test the differences of speed between multiple languages: http://code.google.com/p/languagebenchmarks/ By now i only have the implementation with only Python (without external optimizations) and standard C, showing that python is only between 20 and 40 times slower than C and a promedium of 35, which is not much. I want to add the code generated by pypy to compare how good is optimiced in relation to C and Python, and in the future to other languages. The algorithm is very linear and doesnt make use of threads or anything too much complicated. It does extensive use of lists and strings. But when running the translator with the main.py file i have an error in an initial line: #python pypy-dist/pypy/translator/goal/translate.py main.py File "main.py", line 6, in N_entradas = int(sys.argv[1]) ValueError: invalid literal for int() with base 10: 'main.py' The program is called with 2 integers determinning the number of bits of input and the number of bits of output in an electronic true table which the algoritm should optimice with the QuineMcClusky algorithm. The true table is passed through stdin. By eample: ./main.py 3 2 00010 00101 01011 0111- 100-- 101-1 11000 11100 That is a random table of example. An electronic should understand that, but the fact is that the algoritm have to convert that in a set of lists of string and work with them iterativelly, until it gets the best possible optimization. What i understand of the error line, is that it cant convert 'main.py' to integer. So seems like argv is like ['python pypy-dist/pypy/.....','main.py'] Does the translator haves to run the code to translate it? The 'main.py' should be the first argument received by the program not the second one. How can i pass any argument in the right order? Thanks in advance. Diego. From nshepperd at gmail.com Mon Jul 21 08:41:47 2008 From: nshepperd at gmail.com (Neil Shepperd) Date: Mon, 21 Jul 2008 16:41:47 +1000 Subject: [pypy-dev] I have a problem interpreting pypy functionality. In-Reply-To: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> References: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> Message-ID: <1216622507.6622.17.camel@neil> Hi, On Sat, 2008-07-19 at 00:00 -0300, Diego Jacobi wrote: > What i understand of the error line, is that it cant convert 'main.py' > to integer. > So seems like argv is like ['python pypy-dist/pypy/.....','main.py'] > > Does the translator haves to run the code to translate it? > The 'main.py' should be the first argument received by the program not > the second one. > > How can i pass any argument in the right order? > > Thanks in advance. > Diego. The pypy translator is not a *module* translator. It takes an entry-point function (equivalent to main() in C) and converts that to C. Take a look at pypy/translator/goal/target-nopstandalone.py to see how it is defined. The important thing is that any code you want to translate into C must be placed in a function (and obviously used by the entry-point function). Code in the module body itself will be executed immediately by the translator (under Python interpreter). > ?In some of the tests in the guide i see that after compiling with > t.compile_c() i can directly use the function in python. The function is converted to C and compiled to an extension module (DLL). This module is saved in a temporary directory somewhere (/tmp). It's probably not a good idea to try to open that file itself, better to just use the (compiled) function returned by compile_c(). Hope I make sense. Neil. From jacobidiego at gmail.com Wed Jul 23 17:59:33 2008 From: jacobidiego at gmail.com (Diego Jacobi) Date: Wed, 23 Jul 2008 12:59:33 -0300 Subject: [pypy-dev] I have a problem interpreting pypy functionality. In-Reply-To: <20080723154921.4ef56425.simon@arrowtheory.com> References: <5de034af0807182000k67d3c0c8r668f53ced81ae5c5@mail.gmail.com> <20080721232918.d1aea981.simon@arrowtheory.com> <5de034af0807212150k6b8e584ei8cdc220814ddff84@mail.gmail.com> <20080723001312.e40d994b.simon@arrowtheory.com> <5de034af0807220903g35268176h8c38fdc7917a7ede@mail.gmail.com> <20080723135210.f4105b9a.simon@arrowtheory.com> <5de034af0807222237m52100bbapb098e7954b1167c3@mail.gmail.com> <20080723154921.4ef56425.simon@arrowtheory.com> Message-ID: <5de034af0807230859y5a5ac975o2e95a3ea3dc934ec@mail.gmail.com> 2008/7/23 Simon Burton : > On Wed, 23 Jul 2008 02:37:20 -0300 > "Diego Jacobi" wrote: > >> >> Is something about a negative slice, i replaced all my negative slices >> but the error still saying the same. > > try adding assert(index>=0) statements. > >> >> Is it possible an error of pypy? or it is sure that it is my code. Why >> doesnt point me to the line being parsed? > > It is a "global" error. Very common when compiling rpython. > > Simon. > Thanks that solved the problem, i will have to write lots of asserts. I think that this tells me that i can not continue with this: File "/home/data/diego/Desktop/Comparativa/PyPy/pypy-dist/pypy/annotation/annrpython.py", line 317, in ondegenerated raise AnnotatorError(msgstr) pypy.tool.error.AnnotatorError: annotation of v0 degenerated to SomeObject() v0 = getattr(terms_0, ('sort')) Happened at file /home/data/diego/Desktop/Comparativa/PyPy/QuineMcClusky.py line 51 ==> terms.sort( __sort_condition ) marks = [ False for x in range(len(ones)) ] .. SomeObject() origin: block at 240 op=0 Previous annotation: (none) There was previusly a lambda function on sort, i changed that for a normal function but the error is the same. If i am not wrong that means that sorting is not supported. Cheers. PD: i recently noted that i am sending thoose mails directly to you, sorry. From chris at simplistix.co.uk Fri Jul 25 18:15:41 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Fri, 25 Jul 2008 17:15:41 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> Message-ID: <4889FC2D.3000609@simplistix.co.uk> Hi All, Martijn Faassen wrote: > I think it's unlikely path to try to make pypy-sandbox, a special > version of PyPy, work in a Zope 2 context right now. I'm inclined to agree, but I'd avoid writing it off until the PyPy guys have agreed. I would be surprised if they can pull something out of their box of tricks here, we may just need to find the right way to phrase the problem ;-) At the very least, we should be trying to find the "right" way for realms beyond Zope 2, which it sounds like both myself and Stefan are encountering... > I think there are two possible paths: > > * try to patch up RestrictedPython by closing any new Python 2.5 > related holes. This would require figuring out what these are. This has been "the plan", it has, unfortunately, met prettymuch universal derision from the PyPy guys and silence from Jim :-( > * reimplement RestrictedPython using zope 3 security proxies This is only half the problem, though... > Zope 3 security proxies right now do the following: > > 1) control any attribute access/setting by asking a security policy > whether it's allowed > > 2) if you call a method on a proxied object, the result will be proxied as well > > 3) if you pass something into a method on a proxied object, the thing > passed as an argument will be proxied as well. > > I think to implement something like Restricted Python we would need 1 > (with a Zope 2 specific custom security policy) and 2, but not 3. 3 > causes proxies to start spreading through Zope 2 code Why would they cause havoc? They'd only result from stuff in a python script calling back into Zope, surely? It's a shame we can't form a better "blood/brain barrier" that proxies on the way in and de-proxies on the way out... > The hope of this strategy is that it's easier to maintain in the face > of new Python versions than the current RestrictedPython approach. Right, but there's still the language problems, of which importing is the most serious, but I'm sure there are more lurking there... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From holger at merlinux.eu Fri Jul 25 20:46:09 2008 From: holger at merlinux.eu (holger krekel) Date: Fri, 25 Jul 2008 20:46:09 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <4889FC2D.3000609@simplistix.co.uk> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> Message-ID: <20080725184608.GK22324@trillke.net> Hi Chris, Martijn, all, On Fri, Jul 25, 2008 at 17:15 +0100, Chris Withers wrote: > Hi All, > > Martijn Faassen wrote: > > I think it's unlikely path to try to make pypy-sandbox, a special > > version of PyPy, work in a Zope 2 context right now. > > I'm inclined to agree, but I'd avoid writing it off until the PyPy guys > have agreed. I would be surprised if they can pull something out of > their box of tricks here, we may just need to find the right way to > phrase the problem ;-) > > At the very least, we should be trying to find the "right" way for > realms beyond Zope 2, which it sounds like both myself and Stefan are > encountering... Here are some of my thoughts on the topic - Armin, who did most of the current sandbox impl is on vacation like are others btw. There is not too much of explicit "pypy group-think" on the topic yet :) Basically I see packaging and conceptual concerns. For the first, i think we can assume that we can have a easy_install pypy-python-sandbox which would install a nice sandboxed interpreter executable that you could drive from an application via some nice API. You could control what the sandbox sees as files, and its cpu and ram usage. This basically is doable today. What the sandboxed code currently could't easily do is interact with app objects from the driving process. However, I think we could provide means to represent app objects within the sandboxed program and let the parent/app process see and decide about operation on them. (see e.g. transparent proxies here http://codespeak.net/pypy/dist/pypy/doc/objspace-proxies.html#transparent-proxies for some existing tech). This might resemble what Martijn proposed by reimplementing restricted python on top of security proxies. However, you'd probably need to care a lot less regarding controling code paths that lead to IO like crafting a __builtins__ that doesn't have open etc. You could also mindlessly allow the sandboxed code to import python modules, including ones that your app provides for manipulations because you can be sure that no IO is going to escape. And for this you wouldn't need to mess with __import__ but could allow file access to .py files from the standard lib directory plus ones from app directories. I think the remaining crucial point is how to conveniently define and implement a python object protocol for the proxy interaction layer. This is a two-sided question: what is convenient for the driving app and what is safely doable for our generated pypy-sandbox interpreter. best, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From whycode at gmail.com Mon Jul 28 02:11:15 2008 From: whycode at gmail.com (rahul garg) Date: Sun, 27 Jul 2008 18:11:15 -0600 Subject: [pypy-dev] numpy and pypy : info required Message-ID: Hi. I am writing a compiler called unPython (http://code.google.com/p/unpython) a very young and immature compiler for compiling Python to C. I am presenting the compiler at SciPy conference and need to write a paper about it. Under related work, I want to describe PyPy and also the status of NumPy support in PyPy. Can someone comment on whats the status of NumPy support in PyPy? Btw, unPython is a compiler I am writing for my Masters thesis. It compiles Python to C code with particular focus on NumPy. In fact currently, not much of Python is supported. One particular focus of unPython is parallel computing .. in particular OpenMP like parallel-for loops. I will release a new (much better) version over the next 1-2 weeks and will let people here know as well if its not considered spam :) So if you want to try out the compiler, I would request you to wait 1-2 weeks. thanks, rahul -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at simplistix.co.uk Tue Jul 29 00:22:11 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 28 Jul 2008 23:22:11 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> Message-ID: <488E4693.3060403@simplistix.co.uk> Martijn Faassen wrote: >>> * try to patch up RestrictedPython by closing any new Python 2.5 >>> related holes. This would require figuring out what these are. >> This has been "the plan", it has, unfortunately, met prettymuch universal >> derision from the PyPy guys and silence from Jim :-( > > Well, the PyPy people aren't familiar with this code base. It's been > proven enough technology for the last 8 years or so, so I suspect this > plan is at least feasible, if hardly perfect. Well, I dunno. Part of me is trying to get to the bottom of why the PyPy guys hate RestrictedPython so much. It seems a robust way of providing a secured (as opposed to sandboxed) pthon environment and while it requires work when the AST changes, I've yet to see someone suggest a suitable alternative, as this discussion has kindof showed ;-) >> It's a shame we can't form a better "blood/brain barrier" that proxies on >> the way in and de-proxies on the way out... > > Uh, Chris, that was exactly what I was talking about. You could do so, > but you'd need to alter the way proxies work. I'm not sure where you could generically plug in the un-proxying code... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Tue Jul 29 00:24:12 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 28 Jul 2008 23:24:12 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <20080725184608.GK22324@trillke.net> <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> Message-ID: <488E470C.7000903@simplistix.co.uk> Martijn Faassen wrote: > Just for context, the Python 2.5 porting project is happening this > summer as part of the google summer of code. We're talking about a > timeline of weeks here. We're also talking about an important feature > used by *many* for the last 7 or 8 years or so in a stable release of > Zope 2. There's lots of code floating around that uses this feature. > The new code would need to run on the platforms that Zope 2 runs on. Agreed. > perspective. But from the current Zope 2 perspective, this looks like > a rather long-term development path with a lot of bugs to work out. Agreed. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From holger at merlinux.eu Tue Jul 29 10:29:11 2008 From: holger at merlinux.eu (holger krekel) Date: Tue, 29 Jul 2008 10:29:11 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <20080725184608.GK22324@trillke.net> <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> Message-ID: <20080729082911.GB9619@trillke.net> Hi Martijn, Chris, On Mon, Jul 28, 2008 at 11:09 +0200, Martijn Faassen wrote: > On Fri, Jul 25, 2008 at 8:46 PM, holger krekel wrote: > [description of how this might work] > > Cool, thanks for the description. you are welcome! > Just for context, the Python 2.5 porting project is happening this > summer as part of the google summer of code. We're talking about a > timeline of weeks here. We're also talking about an important feature > used by *many* for the last 7 or 8 years or so in a stable release of > Zope 2. There's lots of code floating around that uses this feature. > The new code would need to run on the platforms that Zope 2 runs on. > > If I were to write a new version of sandboxed Python in a new > framework, this sounds potentially useful. I know Chris is interested > in such technology aside from the interest from the Zope 2 > perspective. But from the current Zope 2 perspective, this looks like > a rather long-term development path with a lot of bugs to work out. Indeed, what i described is not likely to come to happen in the next weeks. > The pay-off would be a more secure version of Python Script in Zope 2 > that is hopefully more easy to port to new Python versions. That would > be nice, but the risks involved in this project are also rather huge, > and unfortunately I don't think the pay-off is huge enough to warrant > a lot of investment in time by (rare) Zope 2 hackers. I'd be happy if > one of them proved me wrong, but it doesn't seem to be a good idea to > let the new Python 2.5 compatible version of Zope 2 be dependent on > unfinished PyPy technology.... Makes sense to me as it stands. I am sure that when the time comes we'll find good use uses for a robust virtualized embeddable Python Interpreter. Having people or parties already asking for it (or better yet, investing into it :-) would certainly accelerate it. cheers, holger -- collaborative expert contracting: http://merlinux.eu PyPy Python/Compiler tool chain: http://codespeak.net/pypy pylib py.test/greenlets/svn APIs: http://pylib.org From paskma at gmail.com Tue Jul 29 12:55:33 2008 From: paskma at gmail.com (=?WINDOWS-1252?Q?Marek_Pa=9Aka?=) Date: Tue, 29 Jul 2008 12:55:33 +0200 Subject: [pypy-dev] Unable to translate locks Message-ID: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> Hi, PyPy hackers, I am playing with Python translation to C, especially with threads. I am using start_new_thread function provided by pypy.module.thread.ll_thread. My example (see mytasks.py in attachment) works fine. But when I tried to use locks (provided by the same module), I got an exception during the translation (see mytasks_err.py for the source code and translation_trace.txt). End of the trace: [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/c/database.py", line 156, in getcontainernode [translation:ERROR] node = nodefactory(self, T, container, **buildkwds) [translation:ERROR] File "/home/data/opt/svnpypy/vanilla-dist/pypy/translator/c/node.py", line 860, in opaquenode_factory [translation:ERROR] raise Exception("don't know about %r" % (T,)) [translation:ERROR] Exception: don't know about I have no idea whats the problem, translation of pypy interpreter with thread module enabled works perfectly. Maybe it is caused by the fact that pypy interpreter uses some special annotation policy, is it possible? Please, I would be thankful for any suggestions/comments what to do to make the locks work. Marek Pa?ka -------------- next part -------------- A non-text attachment was scrubbed... Name: mytasks.py Type: text/x-python Size: 338 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: mytasks_err.py Type: text/x-python Size: 403 bytes Desc: not available URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: translation_trace.txt URL: From d.mcneil at qmul.ac.uk Wed Jul 30 00:51:59 2008 From: d.mcneil at qmul.ac.uk (Douglas McNeil) Date: Tue, 29 Jul 2008 23:51:59 +0100 (BST) Subject: [pypy-dev] math calls and errno In-Reply-To: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> References: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> Message-ID: A while back I noticed that some math-related rpython-produced C was much slower than it should have been. After I figured out what was going on, I set it aside, but I see someone mentioned doing some numpy stuff on IRC today so I dug up the tests. Note that the following slowdown only applies to translated code, so ctypes interfaces to numpy itself are immune, and the slowdown doesn't have much effect on pypy-c, so the audience for this is probably limited. Summary: in typical cases the way pypy treats errno causes math calls in rpython to take over a third longer than they should. This can be repaired by (carefully) inlining the errno functions. Details follow. -- A simple rpython target that does nothing but loop and add the results of sin() is much slower than the same code after being translated by shedskin, which gave the same time as the equivalent cython code, which gave the same time as the equivalent handwritten C. def f(): z = 0.0 for i in xrange(100000): for j in xrange(500): z += sin(float(i+j)) return z gcc 4.2.2 (-O3 -fomit-frame-pointer): rpython: 4.160 s shedskin 0.0.28: 3.043 s cython: 3.041 s C: 3.034 s The rpython/C gap persisted with the Intel compiler: icc 10.1 (") rpython: 3.296 s C 2.118 s so clearly the rpython code was doing something that the others weren't which the compilers particularly disliked and the obvious suspect was the error handling. But that should be on the order of a few percent, not an extra third or half. Turns out the errno calls aren't being inlined -- which makes sense, they're external to the implement_*.c files -- and if you force it they're often optimized away. It's quite fragile, thanks to volatility. But replacing the start of pypy_g_ll_math_ll_math_sin with something like volatile int *errno_loc = &errno; block0: l_v591 = (long)(0L); *(errno_loc) = l_v591; l_v590 = sin(l_x_1); l_v589 = *(errno_loc); OP_INT_IS_TRUE(l_v589, l_v593); if (l_v593) { goto block2; } l_v597 = l_v590; goto block1; you get pypy_g_ll_math_ll_math_sin: .L370: pushl %ebx # subl $8, %esp #, call __errno_location # fldl 16(%esp) # l_x_1 movl %eax, %ebx #, D.11955 movl $0, (%eax) #,* D.11955 fstpl (%esp) # call sin # movl (%ebx), %eax #* D.11955, l_v589 testl %eax, %eax # l_v589 je .L372 #, which is at least much improved over the original, and unlike my first few attempts I think this one correctly survives being optimized under both gcc and icc. (Of course what's actually executed in the tests is the version of this which gets inlined into entry_point, but ll_math_sin gets inlined into entry_point in all versions, so that's not causing the difference.) And we're still calling __errno_location like we should. Anyway, now we have: gcc 4.2.2 std rpython 4.160 s gcc 4.2.2 pure C 3.034 s gcc 4.2.2 errno-inlined rpython 3.068 s icc 10.1 std rpython 3.296 s icc 10.1 pure C 2.118 s icc 10.1 errno-inlined rpython 2.244 s and that's much better, especially with gcc. Standard disclaimers apply: this worked on my linux x86 system, with the above compiler versions, under an almost new moon.. and gcc is known for dramatic variations between versions. If it works for anyone else I'd be pleasantly surprised. That said, icc agrees. Unfortunately, as you might expect, this has very modest effects on pypy-c (i.e. barely detectable over the noise, due to the overhead), so I don't know if there will be interest in modifying the C backend to change it. It does improve things considerably for rpythonic math module writers, though. FWIW. Doug, pypy-math-sig member-in-waiting -- Queen Mary College, University of London "Still creating worlds.. Mathematical Sciences, Astronomy Unit .. but now with an accent!" From arigo at tunes.org Wed Jul 30 09:32:52 2008 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 Jul 2008 09:32:52 +0200 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <488E4693.3060403@simplistix.co.uk> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> <488E4693.3060403@simplistix.co.uk> Message-ID: <20080730073252.GA662@code0.codespeak.net> Hi Chris, On Mon, Jul 28, 2008 at 11:22:11PM +0100, Chris Withers wrote: > Well, I dunno. Part of me is trying to get to the bottom of why the PyPy > guys hate RestrictedPython so much. Sorry to have given this impression so much. I can only speak for myself: "hating" is a bit too strong; genereally speaking I have little interest for the "syntactic" side of programming language or language feature implementations (like AST restrictions or manipulations), and I am more interested in solutions that work at the "semantic" level, i.e. at the level of the behavior of objects (like Zope's security proxies, PyPy's various object spaces, and (at a lower level) PyPy's sandboxing). If I have real critics about RestrictedPython it's that it's not Python at all - it's a seriously limited sublanguage. Now I'm sure there are use cases for such a thing, and unlike Python's dead rexec, I can imagine that it can be made safe with enough care. Personally I have not much interested in that (but nothing against people that do). Sorry for the over-reaction. A bientot, Armin. From arigo at tunes.org Wed Jul 30 09:34:54 2008 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 Jul 2008 09:34:54 +0200 Subject: [pypy-dev] Unable to translate locks In-Reply-To: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> References: <4ddb710e0807290355qe2454c8kdcdaa86d3c1fa2f6@mail.gmail.com> Message-ID: <20080730073454.GB662@code0.codespeak.net> Hi, On Tue, Jul 29, 2008 at 12:55:33PM +0200, Marek Pa?ka wrote: > paskma at paskma:goal$ ./translate.py mytasks_err.py I think you need to specify the --thread option: ./translate.py --thread mytasks_err.py A bientot, Armin From chris at simplistix.co.uk Thu Jul 31 17:06:16 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 31 Jul 2008 16:06:16 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20080729082911.GB9619@trillke.net> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <20080725184608.GK22324@trillke.net> <8928d4e90807280209y1a8810b1g11f65f9b8271afd5@mail.gmail.com> <20080729082911.GB9619@trillke.net> Message-ID: <4891D4E8.9090802@simplistix.co.uk> holger krekel wrote: > I am sure that when the time comes we'll find good use uses > for a robust virtualized embeddable Python Interpreter. > Having people or parties already asking for it (or better yet, > investing into it :-) would certainly accelerate it. What sort of investment would be needed for something that worked like RestrictedPython but was implemented in a way that you guys would be happy to support and also had controllable memory and processor usage and that behaved like a normal python library from the end user's point of view? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk From chris at simplistix.co.uk Thu Jul 31 17:15:02 2008 From: chris at simplistix.co.uk (Chris Withers) Date: Thu, 31 Jul 2008 16:15:02 +0100 Subject: [pypy-dev] Zope2-RPython implementation help wanted.. In-Reply-To: <20080730073252.GA662@code0.codespeak.net> References: <20aa8c370807170008y2f375961j72a489377b2f8124@mail.gmail.com> <20080717072508.GF16827@trillke.net> <487EF825.2020704@simplistix.co.uk> <20080717094009.GA22440@trillke.net> <8928d4e90807190701k29f80d57t783db58ba3d7e827@mail.gmail.com> <4889FC2D.3000609@simplistix.co.uk> <8928d4e90807280158p7c28fa7du48b85ef909676db5@mail.gmail.com> <488E4693.3060403@simplistix.co.uk> <20080730073252.GA662@code0.codespeak.net> Message-ID: <4891D6F6.5010607@simplistix.co.uk> Armin Rigo wrote: > feature implementations (like AST restrictions or manipulations), and I > am more interested in solutions that work at the "semantic" level, i.e. > at the level of the behavior of objects (like Zope's security proxies, > PyPy's various object spaces, and (at a lower level) PyPy's sandboxing). Well, this is good to hear :-) > If I have real critics about RestrictedPython it's that it's not Python > at all - it's a seriously limited sublanguage. I wonder how much of this is based on the problem I set in Vilnius? Because of the rough edges of RestrictedPython and the limited resources I had to get the challenge ready, I think the actual problem I set was a lot harder than I meant. Maciej was showing me that in the environment I specified, you couldn't even interate over a tuple of integers :-/ > Now I'm sure there are > use cases for such a thing, In it's normal environment, the use case for Restricted Python is huge: - you don't need to create classes as you get all the objects you could want to manipulate from the result of the Zope environment - it's *really nice* having an environment where you can write python but where all the security checks are done for you without having to do explicit checks: >>> x.a Unauthorized: You are not allowed to access 'a'. - it would be *really really nice* if such an environment prevented you chewing through excess memory and processing power, which RestrictedPython certainly doesn't offer... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk