From arigo at tunes.org Sat Jan 4 12:22:44 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 4 Jan 2014 12:22:44 +0100 Subject: [pypy-dev] Haswell System In-Reply-To: References: <52A876E1.9070302@gmail.com> <52AA2051.1030904@gmail.com> Message-ID: Hi all, (sent for the pypy-dev mailing list) A quick status update about STM. As I've said on IRC, I am *again* looking at a new approach. This would change the core of STM, but lets us keep most of the work we did on top of it, i.e. inside PyPy and PyPy's JIT. The pre-prototypes (called "c5" and "c6") as well as the current draft (called "c7") tell me that it might work, and it might work very well indeed, with a rather minor overhead. I cannot estimate the percentages yet, but it's certainly much better than a 2x slow-down --- more like 20%. The ideas apply better to low numbers of CPUs, which is fine for the most common current-generation machines; it would make using only two threads a big win already. See README.txt for details here: https://bitbucket.org/pypy/stmgc/raw/c7/c7 . Sorry if it's written in a terse style. The basic limitation is that it can only work on 64-bit Linux. If you're interested and by any chance you're near Neuchatel, Switzerland, next Thursday 9th Jan, I'm sure Patrick and Pascal wouldn't oppose you showing up at 11 at the University for this presentation: > """ > PyPy is a Python interpreter written in Python, best known for being > often several times faster than the standard CPython. It is also a > ground for experimenting with language implementation ideas (for > Python and others). One such experiment uses STM to replace the > global interpreter lock, in the now-well-published manner; plus some > language design ideas about how to allow the end programmer to obtain > longer, controlled transaction. > > I'll describe the current ideas for a very cheap STM design that > focuses on this case: it is object-based, with an integrated GC; > transaction lengths from medium to huge (several full GC cycles); no > false conflicts; and focusing on low numbers of cpus for now. The > current ideas seem to allow for a "read barrier" that is not a barrier > at all, but only 2 to 4 cpu instructions that the machine can schedule > in parallel, and that don't prevent regular compiler optimizations. > (This relies on a Linux-only system call, though.) Additionally, this > could potentially be extended with HyTM. > """ A bient?t, Armin. From alex.gaynor at gmail.com Thu Jan 9 07:20:28 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 8 Jan 2014 22:20:28 -0800 Subject: [pypy-dev] SSL in the stdlib Message-ID: Hey all, There are a number of serious security improvements that have gone into the stdlib SSL module in Python 3. For reasons that defy understanding, the CPython maintainers have decided not to backport them to Python 2. I'd like to backport a few of them, starting with: blocking SSLv2 by default. How do people feel about this? There are basically no servers on the internet that use SSLv2, as it's completely broken, so all this does is prevent an attack. The downside is that there'd be no way for a user to turn this off if we do it. This would be a serious security hardening IMO. (Note that this mostly only affects OS X, almost every other platform has had SSLv2 turned off in OpenSSL itself). Any objections? Alex -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 9 09:14:14 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 9 Jan 2014 10:14:14 +0200 Subject: [pypy-dev] SSL in the stdlib In-Reply-To: References: Message-ID: On Thu, Jan 9, 2014 at 8:20 AM, Alex Gaynor wrote: > Hey all, > > There are a number of serious security improvements that have gone into the > stdlib SSL module in Python 3. For reasons that defy understanding, the > CPython maintainers have decided not to backport them to Python 2. > > I'd like to backport a few of them, starting with: blocking SSLv2 by > default. How do people feel about this? > > There are basically no servers on the internet that use SSLv2, as it's > completely broken, so all this does is prevent an attack. The downside is > that there'd be no way for a user to turn this off if we do it. > > This would be a serious security hardening IMO. > > (Note that this mostly only affects OS X, almost every other platform has > had SSLv2 turned off in OpenSSL itself). > > Any objections? > Alex I think this particular change is fine, especially that on modern linux systems, sslv2 is not supported anyway From lightdarkadmin at 163.com Thu Jan 9 14:30:32 2014 From: lightdarkadmin at 163.com (HY) Date: Thu, 9 Jan 2014 21:30:32 +0800 (CST) Subject: [pypy-dev] ctypes create_string_buffer Message-ID: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> import ctypes as C s1= "0"* 1024* 1024* 10 # mem engross add 10M bytes b1= C.create_string_buffer(s1 ) # CPU 1 core 100% and mem engross add 300M bytes ? I don't know! please talk me, thanks very match.... I'm chinese, english is not good, sorry! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Thu Jan 9 17:59:59 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Thu, 09 Jan 2014 16:59:59 +0000 Subject: [pypy-dev] Boost.Python and PyPy -- status? Message-ID: <8738kxt7jk.fsf@tsmithe.net> Hi there, I have written a (C)Python extension using Boost::Python, and would like to get it working on PyPy. I came across [1], but that's probably a bit out of date, and the link given there discussing the status then[2] doesn't work any more. Is there an up-to-date status page? Or, does anyone know more? I don't currently have much time to invest in solving many residual issues myself, but I should do come spring / summer. [1] https://mail.python.org/pipermail/pypy-dev/2010-April/005816.html [2] http://paste.pocoo.org/show/203663/ Regards, Toby Smithe From yury at shurup.com Thu Jan 9 18:45:16 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Thu, 09 Jan 2014 18:45:16 +0100 Subject: [pypy-dev] Boost.Python and PyPy -- status? In-Reply-To: <8738kxt7jk.fsf@tsmithe.net> References: <8738kxt7jk.fsf@tsmithe.net> Message-ID: <1389289516.2811.137.camel@newpride> On Thu, 2014-01-09 at 16:59 +0000, Toby St Clere Smithe wrote: > > I have written a (C)Python extension using Boost::Python, and would > like to get it working on PyPy. I came across [1], but that's probably > a bit out of date, and the link given there discussing the status > then[2] doesn't work any more. Is there an up-to-date status page? Or, > does anyone know more? Did you actually try it? I have found that cpyext has seen huge improvements in the last years. This summer, I've managed to compile some rather complex Cython-generated code with minor manual changes... -- Sincerely yours, Yury V. Zaytsev From mail at tsmithe.net Thu Jan 9 19:05:59 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Thu, 09 Jan 2014 18:05:59 +0000 Subject: [pypy-dev] Boost.Python and PyPy -- status? References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> Message-ID: <87ha9drpx4.fsf@tsmithe.net> Hi Yury, "Yury V. Zaytsev" writes: > Did you actually try it? Hah, no! I wasn't that optimistic! > I have found that cpyext has seen huge improvements in the last years. > This summer, I've managed to compile some rather complex > Cython-generated code with minor manual changes... This certainly sounds promising. I'll give it a go in the next couple of weeks, and report back (unless anyone else pipes up warning me not to). If I can get boost::python to build against PyPy, that will be very pleasing. Cheers, Toby From foogod at gmail.com Thu Jan 9 19:14:18 2014 From: foogod at gmail.com (Alex Stewart) Date: Thu, 9 Jan 2014 10:14:18 -0800 Subject: [pypy-dev] cppyy and callbacks Message-ID: Hi all, So I've been working on trying to develop PyPy bindings for the Bullet physics library (http://bulletphysics.org/) using cppyy. So far, after a bit of an initial learning curve, I was able to put together a configuration that maps almost all of the standard Bullet classes and functions, and worked surprisingly well "out of the box" (with no fixup code or anything), and have even successfully ported a basic "hello world" Bullet example from one of the tutorials to Python. (I have to say, I'm pretty impressed with how smoothly it all works!) The one main issue I'm left with at this point, however, is that for any sort of real application, Bullet makes substantial use of callbacks (both in the form of some global C++ function pointers as well as some abstract "callback" classes with virtual methods intended to be subclassed and overridden). My understanding is that cppyy does not currently support any form of calling Python code from C++ (only the other way around), so this presents a bit of a problem. >From what I've read, there are currently efforts to switch cppyy to being cling-based, and that this change might also allow some of this. I was wondering (a) how far off is this likely to be? (is it a "we'll have a beta next week" sort of thing, or a "we haven't even started yet, and might never get around to it" sort of thing?) and (b) will calling Python from C++ automatically be an option as soon as the cling-based stuff is available, or is that something that would be an additional step to add (and thus take more time) after the basic cling-transition is done? I've been futzing around with a workaround option in the meantime which I think might work: 1. Wrap the C++ function pointers/virtual functions with stub code which calls a C function pointer instead, passing C++ objects as "void *" 2. Write helper C++ functions which can accept "void *" and return a result casted to the appropriate C++ type 3. Use cffi or ctypes to have the C function pointers call back into a Python wrapper function, which then calls the helper conversion functions via cppyy to convert the "void *"s back into the correct cppyy objects, and then calls the actual Python callback function/method with those as arguments. Obviously, this is kinda clunky and involves a fair bit of overhead, which I'd really like to avoid, so I'm also curious if anybody has any other suggestions for better ways to do this sort of thing? Any help would be much appreciated! --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 9 21:29:50 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 9 Jan 2014 22:29:50 +0200 Subject: [pypy-dev] Boost.Python and PyPy -- status? In-Reply-To: <87ha9drpx4.fsf@tsmithe.net> References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> Message-ID: On Thu, Jan 9, 2014 at 8:05 PM, Toby St Clere Smithe wrote: > Hi Yury, > > "Yury V. Zaytsev" writes: >> Did you actually try it? > > Hah, no! I wasn't that optimistic! > >> I have found that cpyext has seen huge improvements in the last years. >> This summer, I've managed to compile some rather complex >> Cython-generated code with minor manual changes... > > This certainly sounds promising. I'll give it a go in the next couple of > weeks, and report back (unless anyone else pipes up warning me not > to). If I can get boost::python to build against PyPy, that will be very > pleasing. > > Cheers, > > Toby Hi I'm less optimistic. Boost::Python uses tons of internal stuff as far as I know. We really discourage people from using it, because it's a) slow b) prone to not behave exactly like CPython and c) be incomplete. Cheers, fijal From rymg19 at gmail.com Thu Jan 9 22:12:28 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 9 Jan 2014 15:12:28 -0600 Subject: [pypy-dev] Boost.Python and PyPy -- status? In-Reply-To: <8738kxt7jk.fsf@tsmithe.net> References: <8738kxt7jk.fsf@tsmithe.net> Message-ID: Every tried Shiboken? My guess is that it'd work better with cpyext thatn Boost::Python, since there's less black magic in the background. On Thu, Jan 9, 2014 at 10:59 AM, Toby St Clere Smithe wrote: > Hi there, > > I have written a (C)Python extension using Boost::Python, and would like > to get it working on PyPy. I came across [1], but that's probably a bit > out of date, and the link given there discussing the status then[2] > doesn't work any more. Is there an up-to-date status page? Or, does > anyone know more? > > I don't currently have much time to invest in solving many residual > issues myself, but I should do come spring / summer. > > [1] https://mail.python.org/pipermail/pypy-dev/2010-April/005816.html > [2] http://paste.pocoo.org/show/203663/ > > Regards, > > Toby Smithe > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Ryan When your hammer is C++, everything begins to look like a thumb. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Thu Jan 9 22:19:57 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Thu, 09 Jan 2014 21:19:57 +0000 Subject: [pypy-dev] Boost.Python and PyPy -- status? References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> Message-ID: <8761pssvia.fsf@tsmithe.net> Hi fijal, Maciej Fijalkowski writes: > I'm less optimistic. Boost::Python uses tons of internal stuff as far > as I know. We really discourage people from using it, because it's a) > slow b) prone to not behave exactly like CPython and c) be incomplete. That's a shame, not least because boost::python is what my extension is written with. It's an elegant library, and exposes the Python API in a nice C++ idiom. The code I'm interfacing with uses a number of C++ features, and boost::python deals with these in a clean way. I'm thinking of attempting a Cython version, but I won't have time for that for at least 6 to 8 months, and it would be a shame to give up what I've got. But I suspect you have reason not to be optimistic: part of the appeal of boost::python is the way that it makes dealing with the differences between C++ and Python object semantics mostly transparent, but because CPython uses ref-counting garbage collection, and so then does the default boost::python object policy, it'll probably struggle with PyPy. Cheers, Toby From mail at tsmithe.net Thu Jan 9 22:25:04 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Thu, 09 Jan 2014 21:25:04 +0000 Subject: [pypy-dev] Boost.Python and PyPy -- status? References: <8738kxt7jk.fsf@tsmithe.net> Message-ID: <871u0gsv9r.fsf@tsmithe.net> Ryan Gonzalez writes: > Every tried Shiboken? My guess is that it'd work better with cpyext thatn > Boost::Python, since there's less black magic in the background. No, I'd not heard of Shiboken, but it does look like an interesting project. As I wrote in my reply to fijal, I quite like boost::python, and as that's what my code is written in (and it took a while to write it!), that's what I'm interested in right now. Nonetheless, I am looking for alternatives (for a later version), so thanks. Best, Toby From rymg19 at gmail.com Thu Jan 9 22:51:09 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 9 Jan 2014 15:51:09 -0600 Subject: [pypy-dev] Boost.Python and PyPy -- status? In-Reply-To: <871u0gsv9r.fsf@tsmithe.net> References: <8738kxt7jk.fsf@tsmithe.net> <871u0gsv9r.fsf@tsmithe.net> Message-ID: Shiboken was written for the official Python Qt bindings(PySide). They actually were originally using Boost::Python, but they switched to Shiboken because the binaries were a lot smaller. Unfortunately, there isn't too much documentation. Here are some links: http://seanfisk.com/pyside-docs/shiboken/ - The docs for Shiboken itself http://seanfisk.com/pyside-docs/apiextractor/ - The docs for the typesystem files that describe the bindings to be made http://lynxline.com/superhybrids-part-2-now-qt-pyside/ http://qt-project.org/wiki/PySide_Binding_Generation_Tutorial http://qt-project.org/wiki/PySide_Shiboken_Tutorial - Short but a tad helpful https://qt.gitorious.org/pyside/shiboken/source/a527dd51e69b80c2d5be3a1d8cd60ab2b2616fa5:tests - The tests. Folders that begin with "lib" are the C++ code that will be bound, the ones ending in "binding" are the actual bindings. Note that one of these is VERY long! https://qt.gitorious.org/pyside/pyside/source/d2a47ab8f27af7e74d34797464da85c128c17c37:PySide - The best(and largest) example of all: PySide itself. On Thu, Jan 9, 2014 at 3:25 PM, Toby St Clere Smithe wrote: > Ryan Gonzalez writes: > > Every tried Shiboken? My guess is that it'd work better with cpyext thatn > > Boost::Python, since there's less black magic in the background. > > No, I'd not heard of Shiboken, but it does look like an interesting > project. As I wrote in my reply to fijal, I quite like boost::python, > and as that's what my code is written in (and it took a while to write > it!), that's what I'm interested in right now. Nonetheless, I am looking > for alternatives (for a later version), so thanks. > > Best, > > Toby > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Ryan When your hammer is C++, everything begins to look like a thumb. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at tsmithe.net Thu Jan 9 23:12:54 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Thu, 09 Jan 2014 22:12:54 +0000 Subject: [pypy-dev] Boost.Python and PyPy -- status? References: <8738kxt7jk.fsf@tsmithe.net> <871u0gsv9r.fsf@tsmithe.net> Message-ID: <87wqi8rehl.fsf@tsmithe.net> Hi Ryan, Ryan Gonzalez writes: > Shiboken was written for the official Python Qt bindings(PySide). They > actually were originally using Boost::Python, but they switched to Shiboken > because the binaries were a lot smaller. Unfortunately, there isn't too > much documentation. Here are some links: > > http://seanfisk.com/pyside-docs/shiboken/ - The docs for Shiboken itself > http://seanfisk.com/pyside-docs/apiextractor/ - The docs for the typesystem > files that describe the bindings to be made > http://lynxline.com/superhybrids-part-2-now-qt-pyside/ > http://qt-project.org/wiki/PySide_Binding_Generation_Tutorial > http://qt-project.org/wiki/PySide_Shiboken_Tutorial - Short but a tad > helpful > https://qt.gitorious.org/pyside/shiboken/source/a527dd51e69b80c2d5be3a1d8cd60ab2b2616fa5:tests > - > The tests. Folders that begin with "lib" are the C++ code that will be > bound, the ones ending in "binding" are the actual bindings. Note that one > of these is VERY long! > https://qt.gitorious.org/pyside/pyside/source/d2a47ab8f27af7e74d34797464da85c128c17c37:PySide > - > The best(and largest) example of all: PySide itself. Thanks! That's all very handy. I'll dive in :) Cheers, Toby From rymg19 at gmail.com Fri Jan 10 00:22:24 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 9 Jan 2014 17:22:24 -0600 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: What about the CINT backend? https://bitbucket.org/pypy/pypy/src/52bbec630c1faec5ea0c1772872411de541d507f/pypy/module/cppyy/test/test_cint.py On Thu, Jan 9, 2014 at 12:14 PM, Alex Stewart wrote: > Hi all, > > So I've been working on trying to develop PyPy bindings for the Bullet > physics library (http://bulletphysics.org/) using cppyy. So far, after a > bit of an initial learning curve, I was able to put together a > configuration that maps almost all of the standard Bullet classes and > functions, and worked surprisingly well "out of the box" (with no fixup > code or anything), and have even successfully ported a basic "hello world" > Bullet example from one of the tutorials to Python. (I have to say, I'm > pretty impressed with how smoothly it all works!) > > The one main issue I'm left with at this point, however, is that for any > sort of real application, Bullet makes substantial use of callbacks (both > in the form of some global C++ function pointers as well as some abstract > "callback" classes with virtual methods intended to be subclassed and > overridden). My understanding is that cppyy does not currently support any > form of calling Python code from C++ (only the other way around), so this > presents a bit of a problem. > > From what I've read, there are currently efforts to switch cppyy to being > cling-based, and that this change might also allow some of this. I was > wondering (a) how far off is this likely to be? (is it a "we'll have a beta > next week" sort of thing, or a "we haven't even started yet, and might > never get around to it" sort of thing?) and (b) will calling Python from > C++ automatically be an option as soon as the cling-based stuff is > available, or is that something that would be an additional step to add > (and thus take more time) after the basic cling-transition is done? > > I've been futzing around with a workaround option in the meantime which I > think might work: > > 1. Wrap the C++ function pointers/virtual functions with stub code which > calls a C function pointer instead, passing C++ objects as "void *" > 2. Write helper C++ functions which can accept "void *" and return a > result casted to the appropriate C++ type > 3. Use cffi or ctypes to have the C function pointers call back into a > Python wrapper function, which then calls the helper conversion functions > via cppyy to convert the "void *"s back into the correct cppyy objects, and > then calls the actual Python callback function/method with those as > arguments. > > Obviously, this is kinda clunky and involves a fair bit of overhead, which > I'd really like to avoid, so I'm also curious if anybody has any other > suggestions for better ways to do this sort of thing? > > Any help would be much appreciated! > > --Alex > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Ryan When your hammer is C++, everything begins to look like a thumb. -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Fri Jan 10 00:31:19 2014 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 10 Jan 2014 00:31:19 +0100 Subject: [pypy-dev] Boost.Python and PyPy -- status? In-Reply-To: <8738kxt7jk.fsf@tsmithe.net> References: <8738kxt7jk.fsf@tsmithe.net> Message-ID: 2014/1/9 Toby St Clere Smithe > Hi there, > > I have written a (C)Python extension using Boost::Python, and would like > to get it working on PyPy. I came across [1], but that's probably a bit > out of date, and the link given there discussing the status then[2] > doesn't work any more. Is there an up-to-date status page? Or, does > anyone know more? > Hi, I just did it again, with Boost 1.55: - Apply the patch in https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/patches/boost.patch - EXCEPT the "using python" part, which should be something like: using python : 2.7 : $(PYPY_HOME)/bin/pypy : $(PYPY_HOME)/include : : : .pypy-22 ; Then I got the test in "libs/python/example/tutorial" work correctly. I get segfaults in other example, investigating. > I don't currently have much time to invest in solving many residual > issues myself, but I should do come spring / summer. > > [1] https://mail.python.org/pipermail/pypy-dev/2010-April/005816.html > [2] http://paste.pocoo.org/show/203663/ > > Regards, > > Toby Smithe > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Fri Jan 10 01:17:55 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 9 Jan 2014 16:17:55 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, > I have to say, I'm pretty impressed with how smoothly it all works! good to hear. > The one main issue I'm left with at this point, however, is that for any > sort of real application, Bullet makes substantial use of callbacks (both > in the form of some global C++ function pointers as well as some abstract > "callback" classes with virtual methods intended to be subclassed and > overridden). My understanding is that cppyy does not currently support any > form of calling Python code from C++ (only the other way around), so this > presents a bit of a problem. Indeed. I've had two occasions (not with Reflex, but with CINT) where I needed to call the other way around: fitting of data distributions with python functions, and callbacks when C++ objects get deleted. I've used a custom piece of rpython for both. Overriding of virtual functions can only be done with a helper class deriving from the callback interface on the C++ side. There's no other (portable) way. For global functions, I've found helper with void* + cffi (the way you've described in your mail) enough. Yes, that isn't satisfactory, but last year the priority has been Cling. Which answers this: > From what I've read, there are currently efforts to switch cppyy to being > cling-based, and that this change might also allow some of this. I was > wondering (a) how far off is this likely to be? (is it a "we'll have a beta > next week" sort of thing, or a "we haven't even started yet, and might > never get around to it" sort of thing?) The status is that we'd way underestimated the amount of work. When working with CINT or Reflex, it was always possible to have something functional, as parts that were not working yet, could be handled gracefully. Not so with Clang: if you don't handle something, it will exit one way or another (as it is supposed to do: a compiler stops on bad input). As such, it all took about a year longer, but since end of November last year, we now have a working bindings generator for Cling, but only on CPython. I've only just started (this week) with the groundwork of getting a back-end for cppyy into shape. > (b) will calling Python from C++ automatically be an option as soon as the > cling-based stuff is available, or is that something that would be an > additional step to add (and thus take more time) after the basic > cling-transition is done? Calling from Cling into Python has always been part of it. From there, creating callable function pointers with a wrapper is not so much work. Derived classes is somewhat more convoluted, but also straightforward (has been done before, after all). (The bigger point of using Clang/LLVM was C++11, though, so that has some priority, although so far it seems to be in a rather good shape.) Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Fri Jan 10 01:22:09 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 9 Jan 2014 16:22:09 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Ryan, > What about the CINT backend? the way we "jit" code with CINT, is by generating a temporary file, then running the dictionary generator over it, and pulling everything through the C++ compiler. Works, but ... Also, the CINT backend carries more dependencies than strictly necessary, for historic reasons. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From foogod at gmail.com Fri Jan 10 00:58:14 2014 From: foogod at gmail.com (Alex Stewart) Date: Thu, 9 Jan 2014 15:58:14 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: I'd looked around a bit but could only find vague references to CINT, and it wasn't even clear to me whether a full CINT backend really existed or it was just a hack/experiment. Is it actually suitable for general-purpose use? If so, I'd certainly be happy to try it.. how would one go about switching to using the CINT backend instead of Reflex? --Alex On Thu, Jan 9, 2014 at 3:22 PM, Ryan Gonzalez wrote: > What about the CINT backend? > https://bitbucket.org/pypy/pypy/src/52bbec630c1faec5ea0c1772872411de541d507f/pypy/module/cppyy/test/test_cint.py > > > On Thu, Jan 9, 2014 at 12:14 PM, Alex Stewart wrote: > >> Hi all, >> >> So I've been working on trying to develop PyPy bindings for the Bullet >> physics library (http://bulletphysics.org/) using cppyy. So far, after >> a bit of an initial learning curve, I was able to put together a >> configuration that maps almost all of the standard Bullet classes and >> functions, and worked surprisingly well "out of the box" (with no fixup >> code or anything), and have even successfully ported a basic "hello world" >> Bullet example from one of the tutorials to Python. (I have to say, I'm >> pretty impressed with how smoothly it all works!) >> >> The one main issue I'm left with at this point, however, is that for any >> sort of real application, Bullet makes substantial use of callbacks (both >> in the form of some global C++ function pointers as well as some abstract >> "callback" classes with virtual methods intended to be subclassed and >> overridden). My understanding is that cppyy does not currently support any >> form of calling Python code from C++ (only the other way around), so this >> presents a bit of a problem. >> >> From what I've read, there are currently efforts to switch cppyy to being >> cling-based, and that this change might also allow some of this. I was >> wondering (a) how far off is this likely to be? (is it a "we'll have a beta >> next week" sort of thing, or a "we haven't even started yet, and might >> never get around to it" sort of thing?) and (b) will calling Python from >> C++ automatically be an option as soon as the cling-based stuff is >> available, or is that something that would be an additional step to add >> (and thus take more time) after the basic cling-transition is done? >> >> I've been futzing around with a workaround option in the meantime which I >> think might work: >> >> 1. Wrap the C++ function pointers/virtual functions with stub code which >> calls a C function pointer instead, passing C++ objects as "void *" >> 2. Write helper C++ functions which can accept "void *" and return a >> result casted to the appropriate C++ type >> 3. Use cffi or ctypes to have the C function pointers call back into a >> Python wrapper function, which then calls the helper conversion functions >> via cppyy to convert the "void *"s back into the correct cppyy objects, and >> then calls the actual Python callback function/method with those as >> arguments. >> >> Obviously, this is kinda clunky and involves a fair bit of overhead, >> which I'd really like to avoid, so I'm also curious if anybody has any >> other suggestions for better ways to do this sort of thing? >> >> Any help would be much appreciated! >> >> --Alex >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > > > -- > Ryan > When your hammer is C++, everything begins to look like a thumb. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Fri Jan 10 08:13:05 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 10 Jan 2014 09:13:05 +0200 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: On Fri, Jan 10, 2014 at 1:58 AM, Alex Stewart wrote: > I'd looked around a bit but could only find vague references to CINT, and it > wasn't even clear to me whether a full CINT backend really existed or it was > just a hack/experiment. Is it actually suitable for general-purpose use? > > If so, I'd certainly be happy to try it.. how would one go about switching > to using the CINT backend instead of Reflex? > > --Alex Hey Alex. On a sidenote - can you please subscribe to pypy-dev so I don't have to authorize every single one of your mail? It's relatively painless and it's also very low traffic, so you won't get much uninteresting stuff. Obviously feel free to unsubscribe once you're done. Cheers, fijal From mail at tsmithe.net Fri Jan 10 12:15:16 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Fri, 10 Jan 2014 11:15:16 +0000 Subject: [pypy-dev] Boost.Python and PyPy -- status? References: <8738kxt7jk.fsf@tsmithe.net> Message-ID: <8738kwjdff.fsf@tsmithe.net> Hi Amaury, Thanks for responding. "Amaury Forgeot d'Arc" writes: > I just did it again, with Boost 1.55: > > - Apply the patch in > > https://bitbucket.org/pypy/pypy/src/default/pypy/module/cpyext/patches/boost.patch > > - EXCEPT the "using python" part, which should be something like: > > using python : 2.7 > : $(PYPY_HOME)/bin/pypy > : $(PYPY_HOME)/include > : : : .pypy-22 ; > > Then I got the test in "libs/python/example/tutorial" work correctly. > I get segfaults in other example, investigating. This does indeed seem promising! Keep me posted :) I'll have a look when I get some free time over the next weeks. It would be very cool to get PyViennaCL[1] working on PyPy. [1] http://viennacl.sourceforge.net/ Cheers, Toby From kennylevinsen at gmail.com Fri Jan 10 14:52:11 2014 From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen) Date: Fri, 10 Jan 2014 14:52:11 +0100 Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> Message-ID: ctypes is not nice to use. If you need C-bindings, please look at CFFI. Sent from my iPhone > On 09/01/2014, at 14.30, HY wrote: > > import ctypes as C > s1= "0"* 1024* 1024* 10 > # mem engross add 10M bytes > b1= C.create_string_buffer(s1 ) > # CPU 1 core 100% and mem engross add 300M bytes ? > I don't know! > please talk me, thanks very match.... > > I'm chinese, english is not good, sorry! > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Jan 10 15:21:07 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 10 Jan 2014 15:21:07 +0100 Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> Message-ID: Hi, On Fri, Jan 10, 2014 at 2:52 PM, Kenny Lasse Hoff Levinsen wrote: > On 09/01/2014, at 14.30, HY wrote: > import ctypes as C > s1= "0"* 1024* 1024* 10 > # mem engross add 10M bytes > b1= C.create_string_buffer(s1 ) > # CPU 1 core 100% and mem engross add 300M bytes ? > I don't know! > please talk me, thanks very match.... Fixed, thanks! It might have been a long-standing issue with memory in ctypes: memory was not promptly released, leading to the apparence of leaks. > ctypes is not nice to use. If you need C-bindings, please look at CFFI. It's true that we usually recommend to look at CFFI instead of using ctypes, but we still look at and fix the most obvious performance problems with ctypes. The goal for us is to avoid having it run horribly more slowly than on CPython. A bient?t, Armin. From alendit at googlemail.com Fri Jan 10 15:38:27 2014 From: alendit at googlemail.com (Dimitri Vorona) Date: Fri, 10 Jan 2014 15:38:27 +0100 Subject: [pypy-dev] pypy-dev Digest, Vol 33, Issue 6 In-Reply-To: References: Message-ID: @Wim Where could I find the current state of the cling backend implementation? Regards, Dimitri. On Fri, Jan 10, 2014 at 12:00 PM, wrote: > Send pypy-dev mailing list submissions to > pypy-dev at python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/pypy-dev > or, via email, send a message with subject or body 'help' to > pypy-dev-request at python.org > > You can reach the person managing the list at > pypy-dev-owner at python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of pypy-dev digest..." > > > Today's Topics: > > 1. Re: cppyy and callbacks (Maciej Fijalkowski) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 10 Jan 2014 09:13:05 +0200 > From: Maciej Fijalkowski > To: Alex Stewart > Cc: PyPy Developer Mailing List > Subject: Re: [pypy-dev] cppyy and callbacks > Message-ID: > < > CAK5idxQMp+phKVsazry6WqeqnoGSWPx5gaGL9wWrPz6fewTGBA at mail.gmail.com> > Content-Type: text/plain; charset=UTF-8 > > On Fri, Jan 10, 2014 at 1:58 AM, Alex Stewart wrote: > > I'd looked around a bit but could only find vague references to CINT, > and it > > wasn't even clear to me whether a full CINT backend really existed or it > was > > just a hack/experiment. Is it actually suitable for general-purpose use? > > > > If so, I'd certainly be happy to try it.. how would one go about > switching > > to using the CINT backend instead of Reflex? > > > > --Alex > > Hey Alex. > > On a sidenote - can you please subscribe to pypy-dev so I don't have > to authorize every single one of your mail? It's relatively painless > and it's also very low traffic, so you won't get much uninteresting > stuff. Obviously feel free to unsubscribe once you're done. > > Cheers, > fijal > > > ------------------------------ > > Subject: Digest Footer > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > > ------------------------------ > > End of pypy-dev Digest, Vol 33, Issue 6 > *************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alendit at googlemail.com Fri Jan 10 15:41:49 2014 From: alendit at googlemail.com (Dimitri Vorona) Date: Fri, 10 Jan 2014 15:41:49 +0100 Subject: [pypy-dev] cppyy and callbacks Message-ID: @Wim Where could I find the current state of the cling backend implementation? Regards, Dimitri. PS. Please, ignore the email with the wrong subject. > On Fri, Jan 10, 2014 at 12:00 PM, wrote: > >> Send pypy-dev mailing list submissions to >> pypy-dev at python.org >> >> To subscribe or unsubscribe via the World Wide Web, visit >> https://mail.python.org/mailman/listinfo/pypy-dev >> or, via email, send a message with subject or body 'help' to >> pypy-dev-request at python.org >> >> You can reach the person managing the list at >> pypy-dev-owner at python.org >> >> When replying, please edit your Subject line so it is more specific >> than "Re: Contents of pypy-dev digest..." >> >> >> Today's Topics: >> >> 1. Re: cppyy and callbacks (Maciej Fijalkowski) >> >> >> ---------------------------------------------------------------------- >> >> Message: 1 >> Date: Fri, 10 Jan 2014 09:13:05 +0200 >> From: Maciej Fijalkowski >> To: Alex Stewart >> Cc: PyPy Developer Mailing List >> Subject: Re: [pypy-dev] cppyy and callbacks >> Message-ID: >> < >> CAK5idxQMp+phKVsazry6WqeqnoGSWPx5gaGL9wWrPz6fewTGBA at mail.gmail.com> >> Content-Type: text/plain; charset=UTF-8 >> >> On Fri, Jan 10, 2014 at 1:58 AM, Alex Stewart wrote: >> > I'd looked around a bit but could only find vague references to CINT, >> and it >> > wasn't even clear to me whether a full CINT backend really existed or >> it was >> > just a hack/experiment. Is it actually suitable for general-purpose >> use? >> > >> > If so, I'd certainly be happy to try it.. how would one go about >> switching >> > to using the CINT backend instead of Reflex? >> > >> > --Alex >> >> Hey Alex. >> >> On a sidenote - can you please subscribe to pypy-dev so I don't have >> to authorize every single one of your mail? It's relatively painless >> and it's also very low traffic, so you won't get much uninteresting >> stuff. Obviously feel free to unsubscribe once you're done. >> >> Cheers, >> fijal >> >> >> ------------------------------ >> >> Subject: Digest Footer >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> >> ------------------------------ >> >> End of pypy-dev Digest, Vol 33, Issue 6 >> *************************************** >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Fri Jan 10 18:49:59 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 10 Jan 2014 09:49:59 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Dimitri, > Where could I find the current state of the cling backend implementation? remember, I only just started (as in: a couple of days ago); the code does not currently compile, let alone run a single test. When it gets there, it'll appear here: https://bitbucket.org/pypy/pypy/branch/reflex-support As for the CPython side of things, our development is in this repository: http://root.cern.ch/gitweb?p=root.git;a=summary http://root.cern.ch/git/root.git the bindings are under bindings/pyroot. You'll find that they're too entangled to use (other than by building/installing the whole project, which could be an option as it's not _that_ bad). More direct access to cling (we've tried to prevent LLVM headers from leaking into user code, so everything is hidden under at least one layer) can all be found in core/meta. For bug reports etc., here's the project JIRA: https://sft.its.cern.ch/jira/browse/ROOT https://sft.its.cern.ch/jira/browse/ROOT/component/10760 Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Fri Jan 10 18:58:50 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 10 Jan 2014 09:58:50 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, > I'd looked around a bit but could only find vague references to CINT, and > it wasn't even clear to me whether a full CINT backend really existed or it > was just a hack/experiment. it's quite alive; in high energy physics, Reflex is only used by mapping Reflex information into CINT, then use it from CINT. Is historic, though, and not recommended in general. > Is it actually suitable for general-purpose use? If you're willing to install all of ROOT? (Or a minimal version anyway?) On the one hand, I'd argue against that; on the other, ROOT is available in many science sections of Linux distro's as well as in MacPorts, so it's not that big of a deal. But also the run-time dependencies increase. Anyway, the Reflex backend is the default precisely b/c it does not add any further dependencies. Also, CINT per se does not provide what you want (the code that allows compiling in extra parts is in ROOT proper). > If so, I'd certainly be happy to try it.. how would one go about switching > to using the CINT backend instead of Reflex? Is documented here, can only be done "builtin": http://doc.pypy.org/en/latest/cppyy_backend.html I never made a standalone libcppyy_backend library for CINT, as I don't expect there to be any use (physicists in HEP use by and large only releases provided by their experiments' software group; and CINT should be on its way out now that we have Cling largely working). Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From foogod at gmail.com Sat Jan 11 02:07:59 2014 From: foogod at gmail.com (Alex Stewart) Date: Fri, 10 Jan 2014 17:07:59 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: > > On a sidenote - can you please subscribe to pypy-dev so I don't have > to authorize every single one of your mail? I'm a bit confused.. I've been subscribed to pypy-dev (and receiving list mail) since some time in August. (I just went to the mailman page for the list and logged in with my password and it seems to know who I am too..) I just tried unsubscribing and resubscribing, so maybe that will help clear things up? --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Sat Jan 11 02:38:23 2014 From: foogod at gmail.com (Alex Stewart) Date: Fri, 10 Jan 2014 17:38:23 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hey Wim, Thanks for all the useful info on the status of things! > Overriding of virtual functions can only be done with a helper class > deriving from the callback interface on the C++ side. There's no other > (portable) way. Yeah, I kinda figured this would be required regardless, and that bit isn't really that onerous.. it was mainly the extra cffi -> Python -> C++ -> Python round-trip I was hoping to be able to avoid somehow.. > For global functions, I've found helper with void* + cffi (the way you've > described in your mail) enough. Yes, that isn't satisfactory, but last > year the priority has been Cling. > Entirely understandable (and the Cling stuff sounds like a better solution all round, so I agree with that priority). At least now I know I'm on the right track for an interim solution, and something better is actually on the way. :) As such, it all took about a year longer, but since end of November last > year, we now have a working bindings generator for Cling, but only on > CPython. I've only just started (this week) with the groundwork of getting > a > back-end for cppyy into shape. Out of curiosity, how much work do you expect the cppyy part of things to be? (I'd offer to help, but I suspect it's all way over my head..) This does also beg another question that's sorta been at the back of my mind for a bit: Since all of the underlying stuff gets done in CPython first, Is there any chance that at some point we might see a version of cppyy for CPython as well (with similar usage and dependencies)? It's not something I really need for this particular project, but it would be really cool if the Bullet library bindings I'm making (and potentially other C++ libraries) could eventually be used from either PyPy or CPython interchangeably.. I never made a standalone libcppyy_backend library for CINT, as I don't > expect there to be any use (physicists in HEP use by and large only > releases > provided by their experiments' software group; and CINT should be on its > way > out now that we have Cling largely working). That's ok, by the sound of things the Cling-based stuff is moving along well and I can probably wait around a bit for that instead. :) Anyway, thanks for the diligent work! I'm really looking forward to seeing the cling-cppyy (and will be happy to test, etc) when it's ready. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From lightdarkadmin at 163.com Sun Jan 12 06:28:17 2014 From: lightdarkadmin at 163.com (HY) Date: Sun, 12 Jan 2014 13:28:17 +0800 (CST) Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> Message-ID: <5efb37cf.9166.14384ec05d9.Coremail.lightdarkadmin@163.com> I test 2f0d7dced731 memory isn't large engross. but at the AMD 5000+ CPU, 5 secs CPU 100%. I think this has a BUG. :) At 2014-01-10 22:21:07,"Armin Rigo" wrote: >Hi, > >On Fri, Jan 10, 2014 at 2:52 PM, Kenny Lasse Hoff Levinsen > wrote: >> On 09/01/2014, at 14.30, HY wrote: >> import ctypes as C >> s1= "0"* 1024* 1024* 10 >> # mem engross add 10M bytes >> b1= C.create_string_buffer(s1 ) >> # CPU 1 core 100% and mem engross add 300M bytes ? >> I don't know! >> please talk me, thanks very match.... > >Fixed, thanks! It might have been a long-standing issue with memory >in ctypes: memory was not promptly released, leading to the apparence >of leaks. > >> ctypes is not nice to use. If you need C-bindings, please look at CFFI. > >It's true that we usually recommend to look at CFFI instead of using >ctypes, but we still look at and fix the most obvious performance >problems with ctypes. The goal for us is to avoid having it run >horribly more slowly than on CPython. > > >A bient?t, > >Armin. From arigo at tunes.org Sun Jan 12 21:53:56 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 12 Jan 2014 21:53:56 +0100 Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: <5efb37cf.9166.14384ec05d9.Coremail.lightdarkadmin@163.com> References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> <5efb37cf.9166.14384ec05d9.Coremail.lightdarkadmin@163.com> Message-ID: Hi, On Sun, Jan 12, 2014 at 6:28 AM, HY wrote: > I test 2f0d7dced731 > memory isn't large engross. but at the AMD 5000+ CPU, 5 secs CPU 100%. > I think this has a BUG. :) Yes :-) You can see why it's slow by following the logic in pure Python in lib_pypy/_ctypes/, maybe by using just pdb. It's because every character is individually converted to a ctypes object and back before being stored into the string buffer. I guess we could add a special case to massively speed up the common "array-of-char.value = string". A bient?t, Armin. From lightdarkadmin at 163.com Mon Jan 13 02:55:19 2014 From: lightdarkadmin at 163.com (HY) Date: Mon, 13 Jan 2014 09:55:19 +0800 (CST) Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> <5efb37cf.9166.14384ec05d9.Coremail.lightdarkadmin@163.com> Message-ID: <5857a3bd.2ce5.143894f67cc.Coremail.lightdarkadmin@163.com> thanks you reply my message. I think the string is pure struct, mybe should special process; want a little of change as soon as possible enablement. I have a little think of a way, whether add shareptr class in ctypes or cffi? this at denseness data exchange, it can large cut down memory copy. At 2014-01-13 04:53:56,"Armin Rigo" wrote: >Hi, > >On Sun, Jan 12, 2014 at 6:28 AM, HY wrote: >> I test 2f0d7dced731 >> memory isn't large engross. but at the AMD 5000+ CPU, 5 secs CPU 100%. >> I think this has a BUG. :) > >Yes :-) You can see why it's slow by following the logic in pure >Python in lib_pypy/_ctypes/, maybe by using just pdb. It's because >every character is individually converted to a ctypes object and back >before being stored into the string buffer. I guess we could add a >special case to massively speed up the common "array-of-char.value = >string". > > >A bient?t, > >Armin. From lightdarkadmin at 163.com Mon Jan 13 02:50:42 2014 From: lightdarkadmin at 163.com (HY) Date: Mon, 13 Jan 2014 09:50:42 +0800 (CST) Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> <5efb37cf.9166.14384ec05d9.Coremail.lightdarkadmin@163.com> Message-ID: <62b9d112.2a91.143894b2d51.Coremail.lightdarkadmin@163.com> thanks you reply my message. I think the string is pure struct, mybe should special process; want a little of change as soon as possible enablement. At 2014-01-13 04:53:56,"Armin Rigo" wrote: >Hi, > >On Sun, Jan 12, 2014 at 6:28 AM, HY wrote: >> I test 2f0d7dced731 >> memory isn't large engross. but at the AMD 5000+ CPU, 5 secs CPU 100%. >> I think this has a BUG. :) > >Yes :-) You can see why it's slow by following the logic in pure >Python in lib_pypy/_ctypes/, maybe by using just pdb. It's because >every character is individually converted to a ctypes object and back >before being stored into the string buffer. I guess we could add a >special case to massively speed up the common "array-of-char.value = >string". > > >A bient?t, > >Armin. From arigo at tunes.org Tue Jan 14 08:12:13 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 14 Jan 2014 08:12:13 +0100 Subject: [pypy-dev] ctypes create_string_buffer In-Reply-To: <5857a3bd.2ce5.143894f67cc.Coremail.lightdarkadmin@163.com> References: <212ff1bc.c67c.14377327878.Coremail.lightdarkadmin@163.com> <5efb37cf.9166.14384ec05d9.Coremail.lightdarkadmin@163.com> <5857a3bd.2ce5.143894f67cc.Coremail.lightdarkadmin@163.com> Message-ID: Hi HY, On Mon, Jan 13, 2014 at 2:55 AM, HY wrote: > thanks you reply my message. > I think the string is pure struct, mybe should special process; > want a little of change as soon as possible enablement. > > I have a little think of a way, > whether add shareptr class in ctypes or cffi? > this at denseness data exchange, it can large cut down memory copy. Sorry, I don't understand you... What I said in more details: "lib_pypy/_ctypes/array.py" line 20, setvalue(): this can be done more efficiently. We can change it. It would improve the speed a lot. It is not complicated, but it needs a new function in the _rawffi module: "_rawffi.string2charp()". A bient?t, Armin. From techtonik at gmail.com Tue Jan 14 09:59:04 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 14 Jan 2014 11:59:04 +0300 Subject: [pypy-dev] Reducing Sphinx doc verbosity on command line help Message-ID: I find that documentation for pip is hard to read, because it contains too much filler whitespace and repeated obvious headings. Take this as example: http://www.pip-installer.org/en/latest/usage.html#pip-show -----[cut now]------------------------- pip show Usage pip show [options] ... Description Show information about one or more installed packages. Options *Show Options:* -f, --files Show the full list of installed files for each package. *Other Options:* - *General Options* Examples 1. Show information about a package: $ pip show sphinx --- Name: Sphinx Version: 1.1.3 Location: /my/env/lib/pythonx.x/site-packages Requires: Pygments, Jinja2, docutils -----[cut proposed]----------------------------- pip show pip show [options] ... Show information about one or more installed packages. Options -f, --files Show the full list of installed files for each package. *Examples* *Show information about a package (html anchor)* $ pip show sphinx --- Name: Sphinx Version: 1.1.3 Location: /my/env/lib/pythonx.x/site-packages Requires: Pygments, Jinja2, docutils -- anatoly t. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jan 14 10:06:47 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Jan 2014 10:06:47 +0100 Subject: [pypy-dev] Reducing Sphinx doc verbosity on command line help In-Reply-To: References: Message-ID: On Tue, Jan 14, 2014 at 9:59 AM, anatoly techtonik wrote: > I find that documentation for pip is hard to read, > because it contains too much filler whitespace > and repeated obvious headings. Hi anatoly. Are you sure this is the mailing list you intended to write to? Cheers, fijal From techtonik at gmail.com Tue Jan 14 11:10:18 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 14 Jan 2014 13:10:18 +0300 Subject: [pypy-dev] Reducing Sphinx doc verbosity on command line help In-Reply-To: References: Message-ID: On Tue, Jan 14, 2014 at 12:06 PM, Maciej Fijalkowski wrote: > On Tue, Jan 14, 2014 at 9:59 AM, anatoly techtonik > wrote: > > I find that documentation for pip is hard to read, > > because it contains too much filler whitespace > > and repeated obvious headings. > > Hi anatoly. > > Are you sure this is the mailing list you intended to write to? > Ouch. Thanks, it was meant for pypa-dev, but fast-typing + autocompletion betrayed me. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jan 14 11:44:51 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 14 Jan 2014 11:44:51 +0100 Subject: [pypy-dev] Reducing Sphinx doc verbosity on command line help In-Reply-To: References: Message-ID: On Tue, Jan 14, 2014 at 11:10 AM, anatoly techtonik wrote: > On Tue, Jan 14, 2014 at 12:06 PM, Maciej Fijalkowski > wrote: >> >> On Tue, Jan 14, 2014 at 9:59 AM, anatoly techtonik >> wrote: >> > I find that documentation for pip is hard to read, >> > because it contains too much filler whitespace >> > and repeated obvious headings. >> >> Hi anatoly. >> >> Are you sure this is the mailing list you intended to write to? > > > Ouch. Thanks, it was meant for pypa-dev, but fast-typing + autocompletion > betrayed me. Eh, maybe people should not create mailing lists of huffman distance of 1 in the first place ;-) From techtonik at gmail.com Tue Jan 14 12:09:23 2014 From: techtonik at gmail.com (anatoly techtonik) Date: Tue, 14 Jan 2014 14:09:23 +0300 Subject: [pypy-dev] Reducing Sphinx doc verbosity on command line help In-Reply-To: References: Message-ID: On Tue, Jan 14, 2014 at 1:44 PM, Maciej Fijalkowski wrote: > On Tue, Jan 14, 2014 at 11:10 AM, anatoly techtonik > wrote: > > On Tue, Jan 14, 2014 at 12:06 PM, Maciej Fijalkowski > > wrote: > >> > >> On Tue, Jan 14, 2014 at 9:59 AM, anatoly techtonik > > >> wrote: > >> > I find that documentation for pip is hard to read, > >> > because it contains too much filler whitespace > >> > and repeated obvious headings. > >> > >> Hi anatoly. > >> > >> Are you sure this is the mailing list you intended to write to? > > > > > > Ouch. Thanks, it was meant for pypa-dev, but fast-typing + autocompletion > > betrayed me. > > Eh, maybe people should not create mailing lists of huffman distance > of 1 in the first place ;-) > In ideal world there is stats.python.org with android client, which along other info contains public ipython notebook that lists names of python lists, huffman opinion about them and some traffic meters and quality/noize forecasts for daily fun. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: From 13581850130 at 126.com Tue Jan 14 14:41:39 2014 From: 13581850130 at 126.com (=?GBK?B?u6q66sHB?=) Date: Tue, 14 Jan 2014 21:41:39 +0800 (CST) Subject: [pypy-dev] bug on fibonacci Message-ID: <24abc43.2dcb2.14390fc7117.Coremail.13581850130@126.com> FYI. bug: run fibonacci() for 10 times and it's much lower since the second time. pypy: Python 2.7.3 (87aa9de10f9c, Nov 24 2013, 20:57:21) [PyPy 2.2.1 with GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.2.79)] on darwin Type "help", "copyright", "credits" or "license" for more information. And now for something completely different: ``and nannanj to you as well'' applematoMacBook-Pro:pypy apple$ cat fib.py ####################################################### import time def fibonacci(n): if n < 2: return n return fibonacci(n - 2) + fibonacci(n - 1) for i in range(10): t1 = time.time() fibonacci(40) print time.time() - t1 ######################################################### applematoMacBook-Pro:pypy apple$ pypy fib.py 6.90701889992 143.203435183 140.465835094 139.754812956 139.877434015 136.691543102 135.03269887 138.482003927 ^CTraceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "fib.py", line 10, in fibonacci(40) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) File "fib.py", line 6, in fibonacci return fibonacci(n - 2) + fibonacci(n - 1) KeyboardInterrupt -- ------------------------------------------------------------------------ ??? Mobile?13581850130 E-mail: huahl at 126.com MSN: kvhua at msn.com ??? ?????????????? ??? ??????32???????B?504A ??? 010?62220515 -------------- next part -------------- An HTML attachment was scrubbed... URL: From n210241048576 at gmail.com Tue Jan 14 23:38:44 2014 From: n210241048576 at gmail.com (Robert Grosse) Date: Tue, 14 Jan 2014 17:38:44 -0500 Subject: [pypy-dev] performance problems with Krakatau Message-ID: Hi, I have been trying to use Pypy to speed up the Krakatau decompiler ( https://github.com/Storyyeller/Krakatau). It is a large, pure Python application with several compute intensive parts, so I thought it would work well. Unfortunately, there is no clear speedup, and Pypy requires several times as much memory as well, making it unusual for larger inputs. For example, decompiling a quarter of ASM, I got the following results (execution time, memory usage) cpython 64 - 62.5s, 102.6kb cpython 32 - 69.2s, 54.5kb pypy 2.1.0 - 106.5s, 277.8kb pypy 2.2.1 - 109.2s, 194.6kb Sometimes, 2.2.1 is faster than 2.1.0, but they're both clearly much worse than CPython. These tests were performed on Windows 7 64bit using the prebuilt 32bit binaries of Pypy. I tested the 32bit version of CPython too, to see if the problem was a lack of 64bit support. However, CPython 32bit also vastly outperformed Pypy. Execution time was measured using time.time(). Memory usage was measured by watching the Windows Resource Manager and recording the peak Private value. Similar patterns were seen in Working Set, etc. I thought the increased memory usage at least might be explained by constant overhead from compiled code or from it not running long enough to trigger full garbage collection. However, Pypy continues to use several times as much memory on much larger examples. Does anyone know what could be going on here? Pypy isn't normally slower than CPython. Is there a way for me to tell what the problem is? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jan 15 14:39:32 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 15 Jan 2014 14:39:32 +0100 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: On Tue, Jan 14, 2014 at 11:38 PM, Robert Grosse wrote: > Hi, I have been trying to use Pypy to speed up the Krakatau decompiler > (https://github.com/Storyyeller/Krakatau). It is a large, pure Python > application with several compute intensive parts, so I thought it would work > well. Unfortunately, there is no clear speedup, and Pypy requires several > times as much memory as well, making it unusual for larger inputs. > > For example, decompiling a quarter of ASM, I got the following results > (execution time, memory usage) > > cpython 64 - 62.5s, 102.6kb > cpython 32 - 69.2s, 54.5kb > pypy 2.1.0 - 106.5s, 277.8kb > pypy 2.2.1 - 109.2s, 194.6kb > > Sometimes, 2.2.1 is faster than 2.1.0, but they're both clearly much worse > than CPython. > > These tests were performed on Windows 7 64bit using the prebuilt 32bit > binaries of Pypy. I tested the 32bit version of CPython too, to see if the > problem was a lack of 64bit support. However, CPython 32bit also vastly > outperformed Pypy. > > Execution time was measured using time.time(). Memory usage was measured by > watching the Windows Resource Manager and recording the peak Private value. > Similar patterns were seen in Working Set, etc. > > I thought the increased memory usage at least might be explained by constant > overhead from compiled code or from it not running long enough to trigger > full garbage collection. However, Pypy continues to use several times as > much memory on much larger examples. > > Does anyone know what could be going on here? Pypy isn't normally slower > than CPython. Is there a way for me to tell what the problem is? Hi. It depends on your workload a lot. If you want us to have a look into it, you need to provide a clear and reproducible way to run a benchmark. Cheers, fijal From n210241048576 at gmail.com Wed Jan 15 14:56:41 2014 From: n210241048576 at gmail.com (Robert Grosse) Date: Wed, 15 Jan 2014 08:56:41 -0500 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: What would be the best way to provide this? On Wed, Jan 15, 2014 at 8:39 AM, Maciej Fijalkowski wrote: > On Tue, Jan 14, 2014 at 11:38 PM, Robert Grosse > wrote: > > Hi, I have been trying to use Pypy to speed up the Krakatau decompiler > > (https://github.com/Storyyeller/Krakatau). It is a large, pure Python > > application with several compute intensive parts, so I thought it would > work > > well. Unfortunately, there is no clear speedup, and Pypy requires several > > times as much memory as well, making it unusual for larger inputs. > > > > For example, decompiling a quarter of ASM, I got the following results > > (execution time, memory usage) > > > > cpython 64 - 62.5s, 102.6kb > > cpython 32 - 69.2s, 54.5kb > > pypy 2.1.0 - 106.5s, 277.8kb > > pypy 2.2.1 - 109.2s, 194.6kb > > > > Sometimes, 2.2.1 is faster than 2.1.0, but they're both clearly much > worse > > than CPython. > > > > These tests were performed on Windows 7 64bit using the prebuilt 32bit > > binaries of Pypy. I tested the 32bit version of CPython too, to see if > the > > problem was a lack of 64bit support. However, CPython 32bit also vastly > > outperformed Pypy. > > > > Execution time was measured using time.time(). Memory usage was measured > by > > watching the Windows Resource Manager and recording the peak Private > value. > > Similar patterns were seen in Working Set, etc. > > > > I thought the increased memory usage at least might be explained by > constant > > overhead from compiled code or from it not running long enough to trigger > > full garbage collection. However, Pypy continues to use several times as > > much memory on much larger examples. > > > > Does anyone know what could be going on here? Pypy isn't normally slower > > than CPython. Is there a way for me to tell what the problem is? > > Hi. > > It depends on your workload a lot. If you want us to have a look into > it, you need to provide a clear and reproducible way to run a > benchmark. > > Cheers, > fijal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jan 15 15:02:49 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 15 Jan 2014 15:02:49 +0100 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: On Wed, Jan 15, 2014 at 2:56 PM, Robert Grosse wrote: > What would be the best way to provide this? link to download would work. If you don't have a hosting space, just send it to me privately in email, I'll put it somewhere (or use dropbox or whatever). *Clear* instructions are crucial though, don't assume we know how to install or use a random package. Cheers, fijal From n210241048576 at gmail.com Wed Jan 15 15:21:08 2014 From: n210241048576 at gmail.com (Robert Grosse) Date: Wed, 15 Jan 2014 09:21:08 -0500 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: I've uploaded it to Mediafire. If that doesn't work, I can find some other place. https://www.mediafire.com/?eylpn794d9peu19 Included is the jar I used for the tests, a diff against the current head and instructions. The diff just removes the deleteUnused call, which I thought might skew results, and changes it to only decompile 1/4 of the files so it doesn't take as long. Instructions: Make sure python 2.7 is installed clone and checkout commit b0929533ffa0bb2b6b5bb55fc4f38da2ab85a870 from https://github.com/Storyyeller/Krakatau.git apply diff (this should just change two lines in decompile.py) create a directory named temp Assuming you cloned Krakatau to the directory Krakatau and the asm jar is in the current directory, run python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar For pypy, it would of course instead be pypy -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar On Wed, Jan 15, 2014 at 9:02 AM, Maciej Fijalkowski wrote: > On Wed, Jan 15, 2014 at 2:56 PM, Robert Grosse > wrote: > > What would be the best way to provide this? > > link to download would work. If you don't have a hosting space, just > send it to me privately in email, I'll put it somewhere (or use > dropbox or whatever). *Clear* instructions are crucial though, don't > assume we know how to install or use a random package. > > Cheers, > fijal > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jan 15 17:34:49 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 15 Jan 2014 17:34:49 +0100 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: On Wed, Jan 15, 2014 at 3:21 PM, Robert Grosse wrote: > I've uploaded it to Mediafire. If that doesn't work, I can find some other > place. > https://www.mediafire.com/?eylpn794d9peu19 > > Included is the jar I used for the tests, a diff against the current head > and instructions. The diff just removes the deleteUnused call, which I > thought might skew results, and changes it to only decompile 1/4 of the > files so it doesn't take as long. > > Instructions: > Make sure python 2.7 is installed > > clone and checkout commit b0929533ffa0bb2b6b5bb55fc4f38da2ab85a870 from > https://github.com/Storyyeller/Krakatau.git > apply diff (this should just change two lines in decompile.py) > > create a directory named temp > > Assuming you cloned Krakatau to the directory Krakatau and the asm jar is in > the current directory, run > python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar > > For pypy, it would of course instead be > pypy -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar > > > On Wed, Jan 15, 2014 at 9:02 AM, Maciej Fijalkowski > wrote: >> >> On Wed, Jan 15, 2014 at 2:56 PM, Robert Grosse >> wrote: >> > What would be the best way to provide this? >> >> link to download would work. If you don't have a hosting space, just >> send it to me privately in email, I'll put it somewhere (or use >> dropbox or whatever). *Clear* instructions are crucial though, don't >> assume we know how to install or use a random package. >> >> Cheers, >> fijal > > hey, I get ClassNotFoundException: java/lang/Object, full traceback http://paste.pound-python.org/show/n5D7PDoyegfohmJFKOnu/ From n210241048576 at gmail.com Wed Jan 15 19:20:16 2014 From: n210241048576 at gmail.com (Robert Grosse) Date: Wed, 15 Jan 2014 13:20:16 -0500 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: Oh sorry, I forgot about that. You need to find the rt.jar from your Java installation and pass the path on the command line. For example, if it's located in C:\Program Files\Java\jre7\lib, you could do python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar -path "C:\Program Files\Java\jre7\lib\rt.jar" Obviously on Linux it will be somewhere else. It shouldn't really matter which version of Java you have since the standard library is pretty stable.. On Wed, Jan 15, 2014 at 11:34 AM, Maciej Fijalkowski wrote: > On Wed, Jan 15, 2014 at 3:21 PM, Robert Grosse > wrote: > > I've uploaded it to Mediafire. If that doesn't work, I can find some > other > > place. > > https://www.mediafire.com/?eylpn794d9peu19 > > > > Included is the jar I used for the tests, a diff against the current head > > and instructions. The diff just removes the deleteUnused call, which I > > thought might skew results, and changes it to only decompile 1/4 of the > > files so it doesn't take as long. > > > > Instructions: > > Make sure python 2.7 is installed > > > > clone and checkout commit b0929533ffa0bb2b6b5bb55fc4f38da2ab85a870 from > > https://github.com/Storyyeller/Krakatau.git > > apply diff (this should just change two lines in decompile.py) > > > > create a directory named temp > > > > Assuming you cloned Krakatau to the directory Krakatau and the asm jar > is in > > the current directory, run > > python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar > > > > For pypy, it would of course instead be > > pypy -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar > > > > > > On Wed, Jan 15, 2014 at 9:02 AM, Maciej Fijalkowski > > wrote: > >> > >> On Wed, Jan 15, 2014 at 2:56 PM, Robert Grosse > > >> wrote: > >> > What would be the best way to provide this? > >> > >> link to download would work. If you don't have a hosting space, just > >> send it to me privately in email, I'll put it somewhere (or use > >> dropbox or whatever). *Clear* instructions are crucial though, don't > >> assume we know how to install or use a random package. > >> > >> Cheers, > >> fijal > > > > > > hey, I get ClassNotFoundException: java/lang/Object, full traceback > http://paste.pound-python.org/show/n5D7PDoyegfohmJFKOnu/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 16 10:50:10 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 16 Jan 2014 10:50:10 +0100 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: On Wed, Jan 15, 2014 at 7:20 PM, Robert Grosse wrote: > Oh sorry, I forgot about that. > > You need to find the rt.jar from your Java installation and pass the path on > the command line. For example, if it's located in C:\Program > Files\Java\jre7\lib, you could do > python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar -path > "C:\Program Files\Java\jre7\lib\rt.jar" > Obviously on Linux it will be somewhere else. It shouldn't really matter > which version of Java you have since the standard library is pretty stable.. Thanks, I'm looking into it. Would you mind if we add Krakatau as a benchmark for our nightlies? From fijall at gmail.com Thu Jan 16 14:51:21 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 16 Jan 2014 14:51:21 +0100 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: Hi Robert. This is going to be a long mail, so bear with me :) The first take away is that pypy warmup is atrocious (that's unimpressive, but you might be delighted to hear I'm working on it right now, except I'm writing this mail). It's quite a bit of work, so it might or might not make it to the next pypy release. We also don't know how well it'll work. The runs that I have now, when running 3 times in the same process look like this (this includes other improvements mentioned later): 46s 32s 29s (cpython takes always 29s) Now, this is far from ideal and we're working on making it better (in fact it's a very useful benchmark), but I can pinpoint some stuff that we will fix and some stuff we won't fix in the near future. One thing that I've already fixed today is loops over tuple when doing x in tuple (so tuple.__contains__). One of the problems with this code is that I don't think it's very efficient. While that's not a good reason to be slower than cpython, it gives you an upper bound on what can be optimized away. Example (from java/structuring.py): new = new if old is None else tuple(x for x in old if x in new) now note that this has a complexity of O(n^2), because you're iterating for all of the one tuple and then for each over all of the elements of the other tuple. Another example: return [x for x in zip(*map(self._doms.get, nodes)) if len(set(x))==1][-1][0] this creates quite a few lists, while all it wants to do is to grab the last one. Those tiny loops are found a bit everywhere. I think more consistent data structures will make it a lot faster on both CPython and PyPy. >From our side, we'll improve generator iterators today and warmup some time in the not-so-near future. Speaking of which - memory consumptions is absolutely atrocious. It's a combination of JIT using too much memory, generator iterators not being cleaned correctly *and* some bug that prevents JIT loops from being freed. we'll deal with all of it, give us some time (that said, the memory consumption *will* be bigger than cpython, but hopefully by not that much). I'm sorry I can't help you as much as I wanted Cheers, fijal On Thu, Jan 16, 2014 at 10:50 AM, Maciej Fijalkowski wrote: > On Wed, Jan 15, 2014 at 7:20 PM, Robert Grosse wrote: >> Oh sorry, I forgot about that. >> >> You need to find the rt.jar from your Java installation and pass the path on >> the command line. For example, if it's located in C:\Program >> Files\Java\jre7\lib, you could do >> python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar -path >> "C:\Program Files\Java\jre7\lib\rt.jar" >> Obviously on Linux it will be somewhere else. It shouldn't really matter >> which version of Java you have since the standard library is pretty stable.. > > Thanks, I'm looking into it. Would you mind if we add Krakatau as a > benchmark for our nightlies? From n210241048576 at gmail.com Thu Jan 16 15:25:04 2014 From: n210241048576 at gmail.com (Robert Grosse) Date: Thu, 16 Jan 2014 09:25:04 -0500 Subject: [pypy-dev] performance problems with Krakatau In-Reply-To: References: Message-ID: Hi, thanks for looking into it! Feel free to use it as a benchmark. I'll also look into the problems you mentioned to see if I can make future versions of Krakatau faster. On Thu, Jan 16, 2014 at 8:51 AM, Maciej Fijalkowski wrote: > Hi Robert. > > This is going to be a long mail, so bear with me :) > > The first take away is that pypy warmup is atrocious (that's > unimpressive, but you might be delighted to hear I'm working on it > right now, except I'm writing this mail). It's quite a bit of work, so > it might or might not make it to the next pypy release. We also don't > know how well it'll work. > > The runs that I have now, when running 3 times in the same process > look like this (this includes other improvements mentioned later): > > 46s 32s 29s (cpython takes always 29s) > > Now, this is far from ideal and we're working on making it better (in > fact it's a very useful benchmark), but I can pinpoint some stuff that > we will fix and some stuff we won't fix in the near future. One thing > that I've already fixed today is loops over tuple when doing x in > tuple (so tuple.__contains__). > > One of the problems with this code is that I don't think it's very > efficient. While that's not a good reason to be slower than cpython, > it gives you an upper bound on what can be optimized away. Example > (from java/structuring.py): > > new = new if old is None else tuple(x for x in old if x in new) > > now note that this has a complexity of O(n^2), because you're > iterating for all of the one tuple and then for each over all of the > elements of the other tuple. > > Another example: > > return [x for x in zip(*map(self._doms.get, nodes)) if > len(set(x))==1][-1][0] > > this creates quite a few lists, while all it wants to do is to grab > the last one. > > Those tiny loops are found a bit everywhere. I think more consistent > data structures will make it a lot faster on both CPython and PyPy. > > From our side, we'll improve generator iterators today and warmup some > time in the not-so-near future. > > Speaking of which - memory consumptions is absolutely atrocious. It's > a combination of JIT using too much memory, generator iterators not > being cleaned correctly *and* some bug that prevents JIT loops from > being freed. we'll deal with all of it, give us some time (that said, > the memory consumption *will* be bigger than cpython, but hopefully by > not that much). > > I'm sorry I can't help you as much as I wanted > > Cheers, > fijal > > > On Thu, Jan 16, 2014 at 10:50 AM, Maciej Fijalkowski > wrote: > > On Wed, Jan 15, 2014 at 7:20 PM, Robert Grosse > wrote: > >> Oh sorry, I forgot about that. > >> > >> You need to find the rt.jar from your Java installation and pass the > path on > >> the command line. For example, if it's located in C:\Program > >> Files\Java\jre7\lib, you could do > >> python -i Krakatau\decompile.py -out temp asm-debug-all-4.1.jar -path > >> "C:\Program Files\Java\jre7\lib\rt.jar" > >> Obviously on Linux it will be somewhere else. It shouldn't really matter > >> which version of Java you have since the standard library is pretty > stable.. > > > > Thanks, I'm looking into it. Would you mind if we add Krakatau as a > > benchmark for our nightlies? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Fri Jan 17 22:12:27 2014 From: foogod at gmail.com (Alex Stewart) Date: Fri, 17 Jan 2014 13:12:27 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: > > For global functions, I've found helper with void* + cffi (the way you've > described in your mail) enough. Ok, so I think I'm really close to having something working now, except for one thing: I can't figure out how to access/pass a void pointer via cppyy? If I declare a C++ function/method as taking a "void *" argument, I can't figure out how to construct a Python object/value to pass to it that it will accept (whatever I try (for example, when trying an int, such as what cppyy.addressof returns), I just get "TypeError: 'CPPInstance' object expected, got 'int' instead").. Is there some special way to construct a cppyy "void *" object or something I'm not aware of? I also noticed when I tried defining a global "void *" variable, like so: void *voidp; That any attempt to access it from Python yields: >>>> cppyy.gbl.voidp Traceback (most recent call last): File "", line 1, in AttributeError: object has no attribute 'voidp' (details: no converter available for '') This feels like I must be missing something obvious here..? --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From foogod at gmail.com Sat Jan 18 22:36:09 2014 From: foogod at gmail.com (Alex Stewart) Date: Sat, 18 Jan 2014 13:36:09 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: > > Ok, so I think I'm really close to having something working now, except > for one thing: I can't figure out how to access/pass a void pointer via > cppyy? > So I actually worked around this problem by not using "void *" at all and passing around intptr_t values instead, which actually works better anyway since I realized I can pass them directly to/from cppyy.bind_object and cppyy.addressof (which makes the code come out much cleaner than I'd originally hoped, actually).. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From russel at winder.org.uk Sun Jan 19 18:28:10 2014 From: russel at winder.org.uk (Russel Winder) Date: Sun, 19 Jan 2014 17:28:10 +0000 Subject: [pypy-dev] PyD and PyPy [was Boost.Python and PyPy -- status?] In-Reply-To: References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> Message-ID: <1390152490.5003.10.camel@anglides.winder.org.uk> On Thu, 2014-01-09 at 22:29 +0200, Maciej Fijalkowski wrote: [?] > > I'm less optimistic. Boost::Python uses tons of internal stuff as far > as I know. We really discourage people from using it, because it's a) > slow b) prone to not behave exactly like CPython and c) be incomplete. I am delving into PyD just now to get it working with CPython 3 as well as CPython 2. Can you point me towards the best way of trying to make this also work with PyPy and PyPy3? Thanks. -- Russel. ============================================================================= Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder at ekiga.net 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel at winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder From lamalaxmi7 at gmail.com Mon Jan 20 15:58:28 2014 From: lamalaxmi7 at gmail.com (Little Girlie) Date: Mon, 20 Jan 2014 09:58:28 -0500 Subject: [pypy-dev] Your message on craigslist Message-ID: An HTML attachment was scrubbed... URL: From blackstrat at tampabay.rr.com Mon Jan 20 19:24:09 2014 From: blackstrat at tampabay.rr.com (Little Girlie) Date: Mon, 20 Jan 2014 13:24:09 -0500 Subject: [pypy-dev] Hey there! Message-ID: An HTML attachment was scrubbed... URL: From ladyo54 at yahoo.com Tue Jan 21 00:09:07 2014 From: ladyo54 at yahoo.com (Little Girlie) Date: Mon, 20 Jan 2014 18:09:07 -0500 Subject: [pypy-dev] Expected e-mail Message-ID: <1F162C49.E4A1D308@yahoo.com> An HTML attachment was scrubbed... URL: From sikender at gmail.com Tue Jan 21 06:23:15 2014 From: sikender at gmail.com (Little Girlie) Date: Tue, 21 Jan 2014 00:23:15 -0500 Subject: [pypy-dev] Welcome! Message-ID: An HTML attachment was scrubbed... URL: From sikender at gmail.com Tue Jan 21 10:49:11 2014 From: sikender at gmail.com (Little Girlie) Date: Tue, 21 Jan 2014 03:49:11 -0600 Subject: [pypy-dev] Your message on craigslist Message-ID: <969C7BBA.499D6358@gmail.com> An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue Jan 21 11:34:21 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 21 Jan 2014 11:34:21 +0100 Subject: [pypy-dev] pypyjit/test_pypy_c Message-ID: Hi all, There are failures in pypyjit/test_pypy_c/test_string.py that shows that the JIT gets much worse code in some cases. This is probably related to the refactor-str-types branch merge (at least the dates match). I found and fixed one issue, which was that some code in stringmethods.py has a loop in the slow path, preventing the JIT from inlining the fast path. There are still failures though. I suppose I should mention again that merging big branches without checking all tests first is a Bad Thing to do. A bient?t, Armin. From sikender at gmail.com Tue Jan 21 15:10:25 2014 From: sikender at gmail.com (Little Girlie) Date: Tue, 21 Jan 2014 08:10:25 -0600 Subject: [pypy-dev] Howdy! Message-ID: An HTML attachment was scrubbed... URL: From crazyj_10 at yahoo.com Tue Jan 21 19:07:37 2014 From: crazyj_10 at yahoo.com (Little Girlie) Date: Tue, 21 Jan 2014 13:07:37 -0500 Subject: [pypy-dev] Salutations! Message-ID: <47C85D67.85C64A86@yahoo.com> An HTML attachment was scrubbed... URL: From crazyj_10 at yahoo.com Tue Jan 21 23:34:39 2014 From: crazyj_10 at yahoo.com (Little Girlie) Date: Tue, 21 Jan 2014 16:34:39 -0600 Subject: [pypy-dev] I like your profile Message-ID: <1D4E66A6.FFC12524@yahoo.com> An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Wed Jan 22 03:11:29 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Tue, 21 Jan 2014 18:11:29 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, sorry for not responding earlier; had a bit of rough week at work. > So I actually worked around this problem by not using "void *" at all and > passing around intptr_t values instead, Yes, I was going to suggest that. :) But I'll first start implementing void* now. Later, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From johan.rade at gmail.com Wed Jan 22 08:01:31 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Wed, 22 Jan 2014 08:01:31 +0100 Subject: [pypy-dev] PyPy 2 unicode class Message-ID: At the Leysin Sprint Armin outlined a new design of the PyPy 2 unicode class. He gave two versions of the design: A: unicode with a UTF-8 implementation and a UTF-32 interface. B: unicode with a UTF-8 implementation, a UTF-16 interface on Windows and a UTF-32 interface on UNIX-like systems. (Armin claims that this design can be implemented efficiently.) Armin asked me (a Windows developer) which design I prefer. My answer is A. Why? Short answer: Because surrogate pairs are a pain in the ... ------- Long answer -------- ** A well-designed interface should accurately model the problem domain. ** In this case the problem domain contains the following abstract objects: 1. Unicode code points (i.e. integers in [0, 0x110000)) 2. Unicode characters 3. Unicode strings The relations between these abstract objects are: * There is a 1-1 correspondence between Unicode code points and Unicode characters. * Unicode strings are abstract sequences of Unicode characters. With a UTF-32 interface there is a 1-1 correspondence between these abstract objects and the following concrete Python 2 objects: 1'. int objects n with 0 <= n < 0x110000 2'. unicode objects s with len(s) == 1 3'. unicode objects With a UTF-16 interface there is not a 1-1 correspondence between 2 and 2'. Then we are not modelling the problem domain any more, and we have to deal with nonsense such as: >>> s = u'\N{PHAISTOS DISC SIGN FLUTE}' >>> len(s) 2 Next, would such a change break any existing Python 2 code on Windows? Yes it will. For instance the following code for counting characters in a string: f = [0] * (1 << 16) for c in s: f[ord(c)] += 1 On the other hand, some broken code will be fixed by this change. There is a lot of Windows code that does not handle surrogate pairs correctly. For instance, in an earlier version of Notepad, you had to press Delete twice to delete a character with Unicode code point >= 0x10000. Such broken code might be fixed by this change. I believe that fixing broken code will be more common than breaking working code, so the overall effect on existing code will be beneficial. ---------------------- Finally, I very much dislike the term UTF-32 interface. You can, and should, explain that interface without mentioning encodings at all; just describe the problem domain and how the interface models the problem domain. Encodings should be an implementation detail, and should not leak through the interface at all. Python should make you think in terms of abstractions, not in terms of bytes. Python is not C. Let's call it the natural interface. (I hope this makes more sense than my ramblings on IRC last night.) From me at manueljacob.de Tue Jan 21 13:50:03 2014 From: me at manueljacob.de (Manuel Jacob) Date: Tue, 21 Jan 2014 13:50:03 +0100 Subject: [pypy-dev] =?utf-8?q?pypyjit/test=5Fpypy=5Fc?= In-Reply-To: References: Message-ID: On 2014-01-21 11:34, Armin Rigo wrote: > Hi all, > > There are failures in pypyjit/test_pypy_c/test_string.py that shows > that the JIT gets much worse code in some cases. This is probably > related to the refactor-str-types branch merge (at least the dates > match). I found and fixed one issue, which was that some code in > stringmethods.py has a loop in the slow path, preventing the JIT from > inlining the fast path. There are still failures though. I suppose I > should mention again that merging big branches without checking all > tests first is a Bad Thing to do. Hi, yes, it's right that the merging of refactor-str-types has broken three tests in pypy.module.pypyjit.test_pypy_c.test_string. I didn't know that there are tests in pypyjit/test_pypy_c that are not run by own-linux-*, so I didn't run pypy-c-jit-linux-*. Normally I check the buildbots on the next day to see if something is broken I didn't notice before. But test_decode_ascii() in the same test class is broken since over one month now. So let's come to the following point: I'm in favor of merging branches only if all tests pass. What's preventing me from doing this is that the buildbots are broken in the general case. That makes it difficult to track which tests are broken by the branch or broken in default, too. As soon as one developers doesn't care about the buildbots and several tests fail, the whole continuous testing doesn't work (at least for me ;)). What can we do to prevent that tests are failing for more than one month? As I mentioned on IRC a few days ago, the technical solution for this is to run the buildbots on every commit. Since the test suite takes a lot of time and we have limited resources, we can't do that. So we have to solve this in a non-technical way. Everyone should feel responsible for tests that fail and check whether he broke that test by accident. If not, he should feel free to blame others on IRC or the mailing list. -Manuel From cfbolz at gmx.de Wed Jan 22 12:28:25 2014 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Wed, 22 Jan 2014 12:28:25 +0100 Subject: [pypy-dev] [CFP] Dyla'14, Workshop on Dynamic Languages and Applications, at PLDI, Edinburgh Message-ID: <52DFAB59.4060309@gmx.de> [CFP] Dyla'14, Workshop on Dynamic Languages and Applications, at PLDI, Edinburgh - What: 8th Workshop on Dynamic Languages and Applications. - Where: Co-located with PLDI'14, Edinburgh, UK - When: June 12th, Sponsored by ACM SIGPLAN Submission deadline: March 15th More info on the website: http://www.lifl.fr/dyla14/ Dyla is a place where developers and researchers can discuss new advances in the design, implementation and application of dynamically-typed languages. The expected audience of this workshop includes practitioners and researchers sharing the same interest in dynamically-typed languages. Lua, Python, Ruby, JavaScript and others are gaining a significant popularity both in industry and academia. Nevertheless, each community has the tendency to only look at what it produces. Broadening the scope of each community is the goal of the workshop. To achieve this goal Dyla's program and organization committees are composed of leading persons from many such languages. Topics -- - live programming - programming language extensions - programming environment extensions - domain-specific languages & tooling - executing environments - static & dynamic analyses - meta-object protocols - optional type-checking - reverse engineering - testing environments Organizing committee: -- - Damien Cassou, University of Lille 1, FR - Carl Friedrich Bolz, King's College London, GB - Johan Andersson, Burtcorp in Gothenburg, SE - Roberto Ierusalimschy, Catholic Univ. in Rio de Janeiro, BR - Tom Van Cutsem, Vrije Universiteit Brussel, BE Program committee: -- - Anne Etien, University Lille 1, France - David Schneider, Heinrich-Heine-Universit?t D?sseldorf, DE - Didier Verna, EPITA/LRDE, France - Edd Barrett, Department of Informatics, King's College London, GB - Joe Gibbs Politz, Brown University, USA - Peng Wu, IBM T.J. Watson Research Center, USA - Tim Felgentreff, Hasso-Plattner-Institut, DE - T. Stephen Strickland, University of Maryland, US - Yoshiki Oshima, Viewpoints Research Institute, USA - Zachary P. Beane, Portland, USA - the 5 workshop organizers Abstract: -- Java and C# have been a major influence in the adoption of object-oriented language characteristics: academic features like interfaces, garbage collection, and meta-programming became technologies generally accepted by the industry. However, with the adoption of these languages, their limitations became apparent, as testified by industry reactions: invokedynamic has been included in the latest Java virtual machine release; the dynamic language runtime (DLR) is gaining popularity; C# adopted dynamic as a valid static type. Researchers and practitioners struggle with static type systems, overly complex abstract grammars, simplistic concurrency mechanisms, limited reflection capabilities, and the absence of higher-order language constructs such as delegation, closures, and continuations. Dynamic languages such as Ruby, Python, JavaScript and Lua are a step forward in addressing these problems and are getting more and more popular. To make these languages mainstream, practitioners have to look back and pick mechanisms up in existing dynamic languages such as Lisp, Scheme, Smalltalk and Self. The goal of this workshop is to act as a forum where practitioners can discuss new advances in the design, implementation and application of dynamically-typed languages that, sometimes radically, diverge from the statically typed class-based mainstream. Another objective is to discuss new as well as older "forgotten" languages and features in this context. Format and Submission Information -- The workshop will have a demo-oriented style. The idea is to allow participants to demonstrate new and interesting features and discuss what they feel is relevant for the dynamic-language community. To participate in the workshop, you can either - submit (before March 15th 2014) an article (ACM Tighter Alternate style) describing your presentation and/or tool. Your article, which must include from 2 to 15 pages, will be carefully reviewed by the program committee. If accepted, your article will be presented during the workshop and be published to the ACM Digital Library (at your option) and the workshop's web site. Please submit to http://www.easychair.org/conferences/?conf=dyla14. - or give a 10-minute lightning demo of your work. A dedicated session will be allocated for this, provided there is ample time available. In this case, send us the title of your demo. A session on pair programming is also planned. People will then get a chance to share their technologies by interacting with other participants. From ron.reiter at gmail.com Wed Jan 22 14:08:51 2014 From: ron.reiter at gmail.com (Ron Reiter) Date: Wed, 22 Jan 2014 15:08:51 +0200 Subject: [pypy-dev] Memory leaks in Pypy Message-ID: Something really strange is happening to me. I've stumbled upon a memory leak that happens only when answering a POST request using Twisted and Pypy. The problem is described here: http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak It could either be a bug in the Python code that handles the request and leaks, a but in Twisted, or in Pypy. I've been investing the problem for weeks now, with the following conclusion: * CPython does not leak with this code * Tornado with Pypy does not leak with this code * Answering GET requests instead of POST requests does not leak with this code * I tried switching GCs, collecting using gc.collect and setting PYPY_GC_MAX, those attempts did not help * This happens both on Linux and Mac Insights as to how to debug this are very welcome. Thanks, -- Ron Reiter [image: Facebook] [image: Twitter] [image: LinkedIn] Contact me: [image: Skype] ron.reiter -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed Jan 22 18:28:45 2014 From: arigo at tunes.org (Armin Rigo) Date: Wed, 22 Jan 2014 18:28:45 +0100 Subject: [pypy-dev] STM minor collections Message-ID: Hi Remi, I thought about another possible way to handle minor collections in stm-c7. If we synchronize the threads to do minor collections, then we no longer need to force minor collections after each transaction. This is possible if we use a single coordinated nursery instead of one per thread. This is similar to Java GCs, which are multithreaded but still have only one nursery, out of which the threads allocate "blocks" at a time and then subdivide them in a thread-local way (here a "block" can be some small number of pages). After a transaction commits, the committed memory in the nursery can be seen by all threads, and we can unshare pages in order to modify objects. This is fine as long as we wait for all threads to reach a safe point when we do a minor collection --- which could then be done either using some general parallel GC technique, or something more subtle where each thread moves objects that are likely to be their own --- for example, objects that they allocated themselves. The main difference with Java GCs is what to do with pages that have been unshared. I think that reasoning about the total amount of syscalls needed, we can see that we may as well re-share the pages after a minor collection (because the alternative is to skip that page until the next major collection, but then it will have to be re-shared during the next major collection anyway). A bient?t, Armin. From donald at stufft.io Wed Jan 22 18:37:54 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 22 Jan 2014 12:37:54 -0500 Subject: [pypy-dev] Validate SSL/TLS Message-ID: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Hey there! So I?d like SSL/TLS in Python to be better and at the prompting of Alex I?m opening this thread to discuss the possibility of incorporating some of these changes into PyPy. Basically it boils down to is PyPy willing to: a) Have SSL hostname validation back ported to it b) Have that turned on by default ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From donald at stufft.io Wed Jan 22 18:45:09 2014 From: donald at stufft.io (Donald Stufft) Date: Wed, 22 Jan 2014 12:45:09 -0500 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: On Jan 22, 2014, at 12:40 PM, Alex Gaynor wrote: > (a) is a no brainer to me, there's no backwards compatibility concerns here, right? Not with PyPy itself, it?d obviously be a difference from CPython itself (you?d need some sort of boolean toggle to turn it on and off that wouldn?t expect in CPython 2.7) but that could default to False and if someone doesn?t use it they?d get the same behavior as previous versions of PyPy and the same as CPython 2.7. Obviously if they did use it then their code would only run on PyPy. > > I'm +1 on (b) as well, but let's get (a) done first. I think we need to be being proactive in protecting our users, and the fact that CPython's core devs are playing fast and loose with security (particular on Py2) is no excuse. > > Alex > > > On Wed, Jan 22, 2014 at 11:37 AM, Donald Stufft wrote: > Hey there! > > So I?d like SSL/TLS in Python to be better and at the prompting of Alex I?m opening this thread to discuss the possibility of incorporating some of these changes into PyPy. > > Basically it boils down to is PyPy willing to: > > a) Have SSL hostname validation back ported to it > b) Have that turned on by default > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > > > > -- > "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From alex.gaynor at gmail.com Wed Jan 22 18:40:44 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Wed, 22 Jan 2014 11:40:44 -0600 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: (a) is a no brainer to me, there's no backwards compatibility concerns here, right? I'm +1 on (b) as well, but let's get (a) done first. I think we need to be being proactive in protecting our users, and the fact that CPython's core devs are playing fast and loose with security (particular on Py2) is no excuse. Alex On Wed, Jan 22, 2014 at 11:37 AM, Donald Stufft wrote: > Hey there! > > So I?d like SSL/TLS in Python to be better and at the prompting of Alex > I?m opening this thread to discuss the possibility of incorporating some of > these changes into PyPy. > > Basically it boils down to is PyPy willing to: > > a) Have SSL hostname validation back ported to it > b) Have that turned on by default > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jfcaron3 at gmail.com Wed Jan 22 18:51:12 2014 From: jfcaron3 at gmail.com (=?iso-8859-1?Q?Jean-Fran=E7ois_Caron?=) Date: Wed, 22 Jan 2014 09:51:12 -0800 Subject: [pypy-dev] PyPy Status Blog Updates? Message-ID: <44ECFFB6-0BC6-412B-A972-084C68743B82@gmail.com> Hi Devs, I just lurk on the mailing list so I know stuff is happening, but I've noticed that the PyPy Status Blog hasn't been updated in nearly two months, so others might not know what's happening with PyPy these days. Is there enough news to make a post? I'd be personally interested in knowing what's been happening with numpypy, PyPyROOT, and PyGame/PySDL, since those are the libs I use. Jean-Fran?ois From arigo at tunes.org Wed Jan 22 18:56:32 2014 From: arigo at tunes.org (Armin Rigo) Date: Wed, 22 Jan 2014 18:56:32 +0100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: References: Message-ID: Hi Johan, On Wed, Jan 22, 2014 at 8:01 AM, Johan R?de wrote: > (I hope this makes more sense than my ramblings on IRC last night.) All versions you gave make sense as far as I'm concerned :-) But this last one is the clearest indeed. It seems that Python 3 went that way anyway too, and exposes the same "natural" interface on all platforms including Windows. I'm not saying it's a strong argument for us doing the same thing in PyPy *2*, but it's certainly an extra argument for it. I'd be prepared to say that no portable program should crash because it runs on the "wide" version of Python, even if Windows-only programs are not portable in that sense. The argument that it might actually *fix* more programs than it breaks is interesting too. As far as I'm concerned I'd vote for going with it tentatively (i.e. implementing unicodes as utf-8 strings internally, with an indexing cache). If it's really needed we can always add another layer on top of the implementation to give the utf-16 image of the world again. Anyway, it's a trade-off between raw performance (how can the indexing cache be made fast?) and memory usage, with a side note on RPython getting rid of the not-really-reusable unicode type. A bient?t, Armin. From foogod at gmail.com Wed Jan 22 23:44:10 2014 From: foogod at gmail.com (Alex Stewart) Date: Wed, 22 Jan 2014 14:44:10 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: > > sorry for not responding earlier; had a bit of rough week at work. No worries.. I'm used to some lists where people can take a week or more to respond, so this is actually a refreshing change :) But I'll first start implementing void* now. > That would be awesome, if it's not too much trouble.. Using intptr_t is working fairly well for the callback stuff, but there are some other places in the library where void pointers are passed around (for example, assigning "user data" to some types of objects), which I've been trying to figure out how I would deal with if I couldn't use "void*", so it would definitely be useful.. Sorry to be a pest, but I've got one more question: Currently, for the callbacks, I'm converting the C++ object pointer to an intptr_t, and then on the Python side using cppyy.bind_object to convert that intptr_t back to an object of the specified type. I've noticed, however, that if I (for example) call a function which returns an object of BaseClass, but the object being returned actually happens to be a DerivedClass object, cppyy (somehow) figures this out and it comes through as a DerivedClass automatically in Python (so you don't need to cast it). If, however, the same object gets passed back via a callback (and I use cppyy.bind_object(ptr, BaseClass)), it comes out as a BaseClass regardless (so the Python code needs to know it's actually a DerivedClass somehow, and manually cast it). Is there some way I could do something like bind_object, but have it do the automagical "figure out what derived class this actually is and return that instead" behavior that cppyy can apparently do in other places? --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Thu Jan 23 04:13:43 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 22 Jan 2014 19:13:43 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, > That would be awesome, if it's not too much trouble.. well, void* is trouble by definition, but it has to be dealt with. Doing something with it, is actually not that hard, but getting it consistent is. Some code is in, on the reflex-support branch (to be merged once it is documented): void* is represented as an array (I didn't see a PyCObject or PyCapsule outside of cpyext), both for returns and data members. There is a special case cppyy.gbl.nullptr (C++11 style), which is a unique object to represent NULL. ATM, both int(0) and None can also be used in place of nullptr, but the next step will be to remove that (None is void, not NULL; and 0 is archaic). Is also just a matter of being consistent. addressof() has been changed to accept an array, to return its start address as an integer; bind_object() can take both the result of addressof() as well as the array directly. > but there are some other places > in the library where void pointers are passed around (for example, > assigning "user data" to some types of objects), which I've been trying to > figure out how I would deal with if I couldn't use "void*", so it would > definitely be useful.. Passing C++ instances through void* was already supported, and so that should be fine. There was an issue if the code started out with a void* (e.g. from a function call), but that now works as well. (That consistency thing again.) > Currently, for the callbacks, I'm converting the C++ object pointer to an > intptr_t, and then on the Python side using cppyy.bind_object to convert > that intptr_t back to an object of the specified type. I've noticed, > however, that if I (for example) call a function which returns an object of > BaseClass, but the object being returned actually happens to be a > DerivedClass object, cppyy (somehow) figures this out and it comes through > as a DerivedClass automatically in Python (so you don't need to cast it). Yes, this is based on C++'s (limited) RTTI: the dictionary stores class info both by (human readable) class name as well as by typeid. A pre-compiled (or generated, in the case of Cling) cast function to calculate the offset does the rest. (And if the inheritance is non-virtual, the JIT will elide all of that machinery.) > Is there some way I could do something like bind_object, but have it do the > automagical "figure out what derived class this actually is and return that > instead" behavior that cppyy can apparently do in other places? I've added a parameter 'cast' which can be set to True. Note that the call to bind_object already takes another boolean parameter, owns, so better use keyword args. Note that the class name and the pointer value still have to match (was already true before). There is one more caveat: if a void* pointer was first bound to a base class, and is later bound to a derived class (or vice versa), the memory regulator will not work, so any existing python object will not be reused. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Thu Jan 23 04:14:38 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 22 Jan 2014 19:14:38 -0800 (PST) Subject: [pypy-dev] PyPy Status Blog Updates? In-Reply-To: <44ECFFB6-0BC6-412B-A972-084C68743B82@gmail.com> References: <44ECFFB6-0BC6-412B-A972-084C68743B82@gmail.com> Message-ID: Hi Jean-Francois, > I'd be personally interested in knowing what's been happening with numpypy, > PyPyROOT, and PyGame/PySDL, since those are the libs I use. PyPyROOT is not part of PyPy; it's PyPy plus cppyy builtin with the CINT backend and a ROOT.py tailored to it on top. That's how I installed it on AFS and pypy-c renamed to pypyroot to distinguish that mix. Since last, I got TF1 to work, but have been mostly working on ROOT6, as well as having started a new (completely unrelated) project. Dispersion being part of my life. But with ROOT6 now out in beta, I'm trying to get back by working out the Cling backend. As for blog postings, I still have to follow up on a post that I called "Part 1" back in August 2012, so that's rather embarrassing indeed. I'll see what can be done ... Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From wlavrijsen at lbl.gov Thu Jan 23 04:26:58 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Wed, 22 Jan 2014 19:26:58 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, > Out of curiosity, how much work do you expect the cppyy part of things to > be? (I'd offer to help, but I suspect it's all way over my head..) the work it self is mostly transliterating code. Cleaning it up and removing dependencies is a different matter. > This does also beg another question that's sorta been at the back of my > mind for a bit: Since all of the underlying stuff gets done in CPython > first, Is there any chance that at some point we might see a version of > cppyy for CPython as well (with similar usage and dependencies)? That is the plan, as I have some folks here that are demanding it. As a complete aside ... I'm convinced that the version of cppyy with the loadable C-API could just as well be served with something that SWIG could produce. Not with the one that produces CPython extensions, but using the SWIG parser to generate something akin to Reflex dictionaries. That is not so far off as it may seem, as it does most of that already anyway, just cross-sprinkled with Python C-API calls that would need to be removed. The C-API that needs to be filled in for cppyy is really straightforward, and can be implemented incrementally. (I've added a dummy_backend.cxx to allow some tests to proceed even when genreflex is not installed and that proves the point sufficiently.) The API could also be modified, if need be. SWIG is not useful for us (it can't parse our headers), and some .py would still need to be generated (to provide the exact same interface as it does for CPython), but a 'swig -pypy' based on this should not be too hard. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From steve at pearwood.info Thu Jan 23 13:21:55 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 23 Jan 2014 23:21:55 +1100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: References: Message-ID: <20140123122155.GK3915@ando> On Wed, Jan 22, 2014 at 08:01:31AM +0100, Johan R?de wrote: > At the Leysin Sprint Armin outlined a new design of the PyPy 2 unicode > class. He gave two versions of the design: > > A: unicode with a UTF-8 implementation and a UTF-32 interface. > > B: unicode with a UTF-8 implementation, a UTF-16 interface on Windows > and a UTF-32 interface on UNIX-like systems. With a UTF-8 implementation, won't that mean that string indexing operations are O(N) rather than O(1)? E.g. how do you know which UTF-8 byte(s) to look at to get the character at index 42 without having to walk the string from the start? Have you considered the Flexible String Representation from CPython 3.3? http://www.python.org/dev/peps/pep-0393/ Basically, if the largest code point in the string is U+00FF or below, it is implemented using one byte per character (essentially Latin-1); if the largest code point is U+FFFF or below, it is implemented using two bytes per character (essentially UCS-2); otherwise, it is implemented using four bytes per character (UCS-4 or UTF-32). There's more to the FSR, read the PEP for further detail. -- Steven From oscar.j.benjamin at gmail.com Thu Jan 23 14:27:50 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Thu, 23 Jan 2014 13:27:50 +0000 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: References: Message-ID: <20140123132746.GA3553@gmail.com> On Wed, Jan 22, 2014 at 06:56:32PM +0100, Armin Rigo wrote: > Hi Johan, > > On Wed, Jan 22, 2014 at 8:01 AM, Johan R?de wrote: > > (I hope this makes more sense than my ramblings on IRC last night.) > > All versions you gave make sense as far as I'm concerned :-) But this > last one is the clearest indeed. > > It seems that Python 3 went that way anyway too, and exposes the same > "natural" interface on all platforms including Windows. > > I'm not saying it's a strong argument for us doing the same thing in > PyPy *2*, but it's certainly an extra argument for it. I'd be > prepared to say that no portable program should crash because it runs > on the "wide" version of Python, even if Windows-only programs are not > portable in that sense. The argument that it might actually *fix* > more programs than it breaks is interesting too. The distinction between narrow and wide builds of CPython is just a bug in CPython. Thankfully it is fixed as of CPython 3.3. PyPy should definitely not try to emulate that behaviour. > > As far as I'm concerned I'd vote for going with it tentatively (i.e. > implementing unicodes as utf-8 strings internally, with an indexing > cache). If it's really needed we can always add another layer on top > of the implementation to give the utf-16 image of the world again. > Anyway, it's a trade-off between raw performance (how can the indexing > cache be made fast?) and memory usage, with a side note on RPython > getting rid of the not-really-reusable unicode type. Steven wrote: > With a UTF-8 implementation, won't that mean that string indexing > operations are O(N) rather than O(1)? E.g. how do you know which UTF-8 > byte(s) to look at to get the character at index 42 without having to > walk the string from the start? With an indexing cache you do this by looking it up in the cache. If you cache the offset of the starting byte for every kth character then indexing can be O(k). Building the index is O(N) where N is the length of the string. If you do this on demand (the first time an index operation is attempted) then you'll often not need to do it. Subsequent indexing operations are O(k) where k is a controllable parameter. So there's a O(N) per-string cost if the string is indexed at least once and an O(k) cost per attempt to index into the string. OTOH using utf-8 internally means that encoding and decoding as utf-8 becomes O(1) instead of O(N) and uses less memory since if you do s=b.decode('utf-8') then s and b can share the same memory buffer. The FSR does this for latin-1 strings but utf-8 strings are more useful. So if you're more likely to decode/encode from/to utf-8 than you are to index into a long string that's a big saving. If the string comes from anything other than utf-8 the indexing cache can be built while decoding (and reencoding as utf-8 under the hood). Oscar From mtasic85 at gmail.com Thu Jan 23 14:44:27 2014 From: mtasic85 at gmail.com (Marko Tasic) Date: Thu, 23 Jan 2014 14:44:27 +0100 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: Hi Ron, Exactly same problem I have discovered yesterday in one of my production servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and tornado==3.2 on ArchLinux updated on daily basis. I do not have more details on this, but I can say only that POST requests takes at least 10x slower then on CPython. However, I measured time spent inside method and its 0.01s, and actual sending of response takes 30s. This is very strange for reason. How I can help on this and send you more details? Regards, Marko Tasic On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: > Something really strange is happening to me. I've stumbled upon a memory > leak that happens only when answering a POST request using Twisted and Pypy. > > The problem is described here: > > > http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak > > It could either be a bug in the Python code that handles the request and > leaks, a but in Twisted, or in Pypy. > > I've been investing the problem for weeks now, with the following > conclusion: > > * CPython does not leak with this code > * Tornado with Pypy does not leak with this code > * Answering GET requests instead of POST requests does not leak with this > code > * I tried switching GCs, collecting using gc.collect and setting > PYPY_GC_MAX, those attempts did not help > * This happens both on Linux and Mac > > Insights as to how to debug this are very welcome. > > Thanks, > > -- > > Ron Reiter > [image: Facebook] [image: Twitter] [image: > LinkedIn] > Contact me: [image: Skype] ron.reiter > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From donald at stufft.io Thu Jan 23 16:17:35 2014 From: donald at stufft.io (Donald Stufft) Date: Thu, 23 Jan 2014 10:17:35 -0500 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: Apologies as I?ve just recently sub?d to pypy-dev so I don?t know it?s normal speed and such. Is there enough here that a patch to do (a) would be a reasonable next step? On Jan 22, 2014, at 12:45 PM, Donald Stufft wrote: > > On Jan 22, 2014, at 12:40 PM, Alex Gaynor wrote: > >> (a) is a no brainer to me, there's no backwards compatibility concerns here, right? > > Not with PyPy itself, it?d obviously be a difference from CPython itself (you?d need some sort of boolean toggle to turn it on and off that wouldn?t expect in CPython 2.7) but that could default to False and if someone doesn?t use it they?d get the same behavior as previous versions of PyPy and the same as CPython 2.7. > > Obviously if they did use it then their code would only run on PyPy. > >> >> I'm +1 on (b) as well, but let's get (a) done first. I think we need to be being proactive in protecting our users, and the fact that CPython's core devs are playing fast and loose with security (particular on Py2) is no excuse. >> >> Alex >> >> >> On Wed, Jan 22, 2014 at 11:37 AM, Donald Stufft wrote: >> Hey there! >> >> So I?d like SSL/TLS in Python to be better and at the prompting of Alex I?m opening this thread to discuss the possibility of incorporating some of these changes into PyPy. >> >> Basically it boils down to is PyPy willing to: >> >> a) Have SSL hostname validation back ported to it >> b) Have that turned on by default >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> >> >> >> -- >> "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) >> "The people's good is the highest law." -- Cicero >> GPG Key fingerprint: 125F 5C67 DFE9 4084 > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From alex.gaynor at gmail.com Thu Jan 23 16:18:49 2014 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Thu, 23 Jan 2014 09:18:49 -0600 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: Yeah, I think writing the patch for the backport (off by default... for now) is a good next step. If you send a PR or create a branch I can review it. Alex On Thu, Jan 23, 2014 at 9:17 AM, Donald Stufft wrote: > Apologies as I?ve just recently sub?d to pypy-dev so I don?t know it?s > normal > speed and such. > > Is there enough here that a patch to do (a) would be a reasonable next > step? > > On Jan 22, 2014, at 12:45 PM, Donald Stufft wrote: > > > On Jan 22, 2014, at 12:40 PM, Alex Gaynor wrote: > > (a) is a no brainer to me, there's no backwards compatibility concerns > here, right? > > > Not with PyPy itself, it?d obviously be a difference from CPython itself > (you?d need some sort of boolean toggle to turn it on and off that wouldn?t > expect in CPython 2.7) but that could default to False and if someone > doesn?t use it they?d get the same behavior as previous versions of PyPy > and the same as CPython 2.7. > > Obviously if they did use it then their code would only run on PyPy. > > > I'm +1 on (b) as well, but let's get (a) done first. I think we need to be > being proactive in protecting our users, and the fact that CPython's core > devs are playing fast and loose with security (particular on Py2) is no > excuse. > > Alex > > > On Wed, Jan 22, 2014 at 11:37 AM, Donald Stufft wrote: > >> Hey there! >> >> So I?d like SSL/TLS in Python to be better and at the prompting of Alex >> I?m opening this thread to discuss the possibility of incorporating some of >> these changes into PyPy. >> >> Basically it boils down to is PyPy willing to: >> >> a) Have SSL hostname validation back ported to it >> b) Have that turned on by default >> >> ----------------- >> Donald Stufft >> PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 >> DCFA >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero > GPG Key fingerprint: 125F 5C67 DFE9 4084 > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > > > ----------------- > Donald Stufft > PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 > DCFA > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero GPG Key fingerprint: 125F 5C67 DFE9 4084 -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 23 17:23:55 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 23 Jan 2014 17:23:55 +0100 Subject: [pypy-dev] PyPy Status Blog Updates? In-Reply-To: <44ECFFB6-0BC6-412B-A972-084C68743B82@gmail.com> References: <44ECFFB6-0BC6-412B-A972-084C68743B82@gmail.com> Message-ID: Hi Jean. Sorry for not keeping everyone updated, we're working on this stuff, hope to write something about pygame soon ;-) On Wed, Jan 22, 2014 at 6:51 PM, Jean-Fran?ois Caron wrote: > Hi Devs, I just lurk on the mailing list so I know stuff is happening, but I've noticed that the PyPy Status Blog hasn't been updated in nearly two months, so others might not know what's happening with PyPy these days. Is there enough news to make a post? > > I'd be personally interested in knowing what's been happening with numpypy, PyPyROOT, and PyGame/PySDL, since those are the libs I use. > > Jean-Fran?ois > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From fijall at gmail.com Thu Jan 23 17:26:34 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 23 Jan 2014 17:26:34 +0100 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: > Something really strange is happening to me. I've stumbled upon a memory > leak that happens only when answering a POST request using Twisted and Pypy. > > The problem is described here: > > > http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak > > It could either be a bug in the Python code that handles the request and > leaks, a but in Twisted, or in Pypy. > > I've been investing the problem for weeks now, with the following > conclusion: > > * CPython does not leak with this code > * Tornado with Pypy does not leak with this code > * Answering GET requests instead of POST requests does not leak with this > code > * I tried switching GCs, collecting using gc.collect and setting > PYPY_GC_MAX, those attempts did not help > * This happens both on Linux and Mac > > Insights as to how to debug this are very welcome. > > Thanks, > Hi. I could not quite get it to leak at all (the memory was not growing), but I can try again. Can we start with updating the example so it's runnable (in the comments JP calderone mentions something that's not obvious to me what it should do). Can you also explain for how long you need to run it? What parameters are there etc? All those things makes it really hard for me to debug. -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 23 17:27:16 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 23 Jan 2014 17:27:16 +0100 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: On Thu, Jan 23, 2014 at 2:44 PM, Marko Tasic wrote: > Hi Ron, > > Exactly same problem I have discovered yesterday in one of my production > servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and tornado==3.2 > on ArchLinux updated on daily basis. I do not have more details on this, > but I can say only that POST requests takes at least 10x slower then on > CPython. However, I measured time spent inside method and its 0.01s, and > actual sending of response takes 30s. This is very strange for reason. > > How I can help on this and send you more details? > > Regards, > Marko Tasic > gunicorn is a bit dodge (it's pypy support is not very official), but I can give it a go - please write some sort of reproducible example and we can take it from there. > > > On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: > >> Something really strange is happening to me. I've stumbled upon a memory >> leak that happens only when answering a POST request using Twisted and Pypy. >> >> The problem is described here: >> >> >> http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak >> >> It could either be a bug in the Python code that handles the request and >> leaks, a but in Twisted, or in Pypy. >> >> I've been investing the problem for weeks now, with the following >> conclusion: >> >> * CPython does not leak with this code >> * Tornado with Pypy does not leak with this code >> * Answering GET requests instead of POST requests does not leak with this >> code >> * I tried switching GCs, collecting using gc.collect and setting >> PYPY_GC_MAX, those attempts did not help >> * This happens both on Linux and Mac >> >> Insights as to how to debug this are very welcome. >> >> Thanks, >> >> -- >> >> Ron Reiter >> [image: Facebook] [image: Twitter] [image: >> LinkedIn] >> Contact me: [image: Skype] ron.reiter >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 23 17:28:57 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 23 Jan 2014 17:28:57 +0100 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: On Thu, Jan 23, 2014 at 4:17 PM, Donald Stufft wrote: > Apologies as I?ve just recently sub?d to pypy-dev so I don?t know it?s > normal > speed and such. > > Is there enough here that a patch to do (a) would be a reasonable next step? > > On Jan 22, 2014, at 12:45 PM, Donald Stufft wrote: > Hi Donald. Since I have absolutely no idea what you're talking about, let's start with: * explain the problem and what CPython devs decided to do * explain the solution and what exactly will it break? From johan.rade at gmail.com Thu Jan 23 17:26:29 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Thu, 23 Jan 2014 17:26:29 +0100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: References: Message-ID: <52E142B5.9060103@gmail.com> On 2014-01-22 08:01, Johan R?de wrote: > Next, would such a change break any existing Python 2 code on Windows? > Yes it will. For instance the following code for counting characters in > a string: > > f = [0] * (1 << 16) > for c in s: > f[ord(c)] += 1 I would like to qualify this statement. Getting rid of the UTF-16 interface for the unicode class in Python 2 on Windows will: * not break any well-written code * fix a lot of poorly written code. (The above code snippet is not well-written code.) --Johan From arigo at tunes.org Thu Jan 23 18:13:41 2014 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Jan 2014 18:13:41 +0100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: <20140123132746.GA3553@gmail.com> References: <20140123132746.GA3553@gmail.com> Message-ID: Hi Oscar, Thanks for explaining the caching in detail :-) On Thu, Jan 23, 2014 at 2:27 PM, Oscar Benjamin wrote: > big saving. If the string comes from anything other than utf-8 the indexing > cache can be built while decoding (and reencoding as utf-8 under the hood). Actually, you need to walk the string even to do "u = s.decode('utf-8')". The reason is that you need to check if the byte string is well-formed UTF-8 or not. So we can build the cache eagerly in all cases, it seems. A bient?t, Armin. From arigo at tunes.org Thu Jan 23 21:02:53 2014 From: arigo at tunes.org (Armin Rigo) Date: Thu, 23 Jan 2014 21:02:53 +0100 Subject: [pypy-dev] pypyjit/test_pypy_c In-Reply-To: References: Message-ID: Hi Manuel, On Tue, Jan 21, 2014 at 1:50 PM, Manuel Jacob wrote: > So we have to solve this in > a non-technical way. Everyone should feel responsible for tests that fail > and check whether he broke that test by accident. If not, he should feel > free to blame others on IRC or the mailing list. I agree, and I'm ready to take the blame when I mess something up :-) ...like test_ztranslation failing in cpyext since yesterday, when I changed something in cpyext... A bient?t, Armin. From foogod at gmail.com Thu Jan 23 21:31:34 2014 From: foogod at gmail.com (Alex Stewart) Date: Thu, 23 Jan 2014 12:31:34 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: > > Some code is in, on the reflex-support branch (to be merged once it is > documented): void* is represented as an array (I didn't see a PyCObject or > PyCapsule outside of cpyext), both for returns and data members. There is > a special case cppyy.gbl.nullptr (C++11 style), which is a unique object to > represent NULL. > (I'm a little unclear what it means to represent "void*" as an array (array of what?), but the rest of cppyy seems to be pretty well thought out, so I'll trust you know what you're doing :) ) Out of curiosity, is there some reason you didn't name that cppyy.gbl.NULL instead? It seems to me that "nullptr" could potentially cause namespace collisions if some library decided to have a function/variable by that name, whereas "NULL" would be consistent with standard C++ naming.. Passing C++ instances through void* was already supported, and so that > should > be fine. There was an issue if the code started out with a void* (e.g. from > a function call), but that now works as well. (That consistency thing > again.) For clarification, the specific use case I'm looking at is objects/functions that use "void *" as an "opaque pointer to application data" that the library doesn't want/need to care about (just pass back to the application sometimes). In this context, it would be really nice if there was some easy way to use "void*" to represent a pure Python object as well (not just C++ instances). Obviously, in pypy objects don't have static addresses, so we can't just use a pointer to the Python object, but I was considering the possibility of taking an index or id() of the object and just casting it into "void*" form, and then converting it back and using it to look up objects in a list/dict somewhere.. (Alternately, it occurs to me now one could define a C++ "proxy" object which could store that sort of lookup information, and then the address of the proxy could be stored in the void*, which might be a little more kosher..) In any case, I wanted to make sure that use case was something that would be feasible in your plan for how void pointers will ultimately work.. --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Jan 23 21:54:14 2014 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Jan 2014 07:54:14 +1100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: <20140123132746.GA3553@gmail.com> References: <20140123132746.GA3553@gmail.com> Message-ID: <20140123205414.GL3915@ando> On Thu, Jan 23, 2014 at 01:27:50PM +0000, Oscar Benjamin wrote: > Steven wrote: > > With a UTF-8 implementation, won't that mean that string indexing > > operations are O(N) rather than O(1)? E.g. how do you know which UTF-8 > > byte(s) to look at to get the character at index 42 without having to > > walk the string from the start? > > With an indexing cache you do this by looking it up in the cache. [...] > So there's a O(N) per-string cost if the string is > indexed at least once and an O(k) cost per attempt to index into the string. Plus a memory cost for the cache, and increased complexity for indexing operations. > OTOH using utf-8 internally means that encoding and decoding as utf-8 becomes > O(1) instead of O(N) and uses less memory since if you do s=b.decode('utf-8') > then s and b can share the same memory buffer. CPython caches the UTF-8 encoding of (some? all?) strings too. It's not clear to me if they are cached lazily or eagerly, but either way, I'm not convinced by this argument that this is useful. In most applications, you'll decode from UTF-8 exactly once, when you read the string in (from a file, or the network, say), and you'll encode it at most once, when you write it out again. Many strings won't ever be encoded to UTF-8. I have difficulty imagining any application which will need to repeatedly encode the same string to UTF-8, where the cost of encoding is more significant than the cost of I/O. One possible exception might be printing to the terminal, say at the REPL, but even then, I don't imagine the time savings are worthwhile. I haven't noticed the REPL in Python 3.3 being particularly more responsive than that in 3.2. If there's a speed-up, it's invisible to the user, but at the cost of using extra memory for the UTF-8 cache. > The FSR does this for latin-1 > strings but utf-8 strings are more useful. So if you're more likely to > decode/encode from/to utf-8 than you are to index into a long string that's a > big saving. But I don't believe that is the case for any realistic use-case. Just about every string operation involves walking the string, which needs to walk the string in characters (code points), not UTF-8 bytes. At the Python layer, string indexing might be relatively uncommon, but at the implementation layer, I think every string operation relies on it. But having said all this, I know that using UTF-8 internally for strings is quite common (e.g. Haskell does it, without even an index cache, and documents that indexing operations can be slow). CPython's FSR has received much (in my opinion, entirely uninformed) criticism from one vocal person in particular for not using UTF-8 internally. If PyPy goes ahead with using UTF-8 internally, I look forward to comparing memory and time benchmarks of string operations between CPython and PyPy. -- Steven From wlavrijsen at lbl.gov Thu Jan 23 22:36:19 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Thu, 23 Jan 2014 13:36:19 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, > (I'm a little unclear what it means to represent "void*" as an array (array > of what?), exactly. A PyCObject is useless by design (but lacking in PyPy AFAICT, other than in cpyext, which use would be slow), as is an array of void. It is the best equivalent I could come up with ... I figured that the only thing I ever wanted from a PyCObject, is to get the C-side address. That is why cppyy.addressof() does provide that for arrays of void now. Beyond that, PyCObject's serve to go from C -> Python -> C, with no work in Python. This void array can do that. The only part missing is an as_cobject (one that returns a cpyext-based, CPython PyCObject), to communicate with other extension libraries that use the CPython C-API. Such a function is needed for all other C++ instances also, so I'll add that (PyROOT has it to allow integration with PyQt, for example). > Out of curiosity, is there some reason you didn't name that cppyy.gbl.NULL > instead? It seems to me that "nullptr" could potentially cause namespace > collisions if some library decided to have a function/variable by that > name, whereas "NULL" would be consistent with standard C++ naming.. Is unlikely, as 'nullptr' is a C++11 keyword. I've been thinking of putting it under 'cppyy' rather than 'cppyy.gbl', but the latter seemed to make more sense (it's not a module feature, but a C++ feature). > For clarification, the specific use case I'm looking at is > objects/functions that use "void *" as an "opaque pointer to application > data" that the library doesn't want/need to care about (just pass back to > the application sometimes). In this context, it would be really nice if > there was some easy way to use "void*" to represent a pure Python object as > well (not just C++ instances). Okay, I understand better now. > Obviously, in pypy objects don't have > static addresses, so we can't just use a pointer to the Python object That is solvable (cppyy can bind functions that take/provide PyObject* as it simply hands over the work to cpyext). See for example test_crossing.py in pypy/module/cppyy/test. The larger problem would be that any randomly given python object does not have a guaranteed layout. > but > I was considering the possibility of taking an index or id() of the object > and just casting it into "void*" form, and then converting it back and > using it to look up objects in a list/dict somewhere.. Alex Pyattaev did this (is in the pypy-dev mail archives, somewhere around early Summer 2012). > In any case, I wanted to make sure that use case was something that would > be feasible in your plan for how void pointers will ultimately work.. Let me know if you see anything missing. "void*" is an interesting puzzle, without many people (that I know of) interested in it. The much bigger pain that we have, is the use of "char*" where the intended usage is "byte*". (And I don't actually have a solution for that.) Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From drsalists at gmail.com Thu Jan 23 22:35:59 2014 From: drsalists at gmail.com (Dan Stromberg) Date: Thu, 23 Jan 2014 13:35:59 -0800 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: References: Message-ID: On Tue, Jan 21, 2014 at 11:01 PM, Johan R?de wrote: > At the Leysin Sprint Armin outlined a new design of the PyPy 2 unicode > class. He gave two versions of the design: Why spend brain cycles on a Pypy unicode class, when you could just move on to Pypy3? The majority of the Python community is on-board with the 2-to-3 transition. From estama at gmail.com Thu Jan 23 21:45:25 2014 From: estama at gmail.com (Elefterios Stamatogiannakis) Date: Thu, 23 Jan 2014 22:45:25 +0200 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: <20140123205414.GL3915@ando> References: <20140123132746.GA3553@gmail.com> <20140123205414.GL3915@ando> Message-ID: <52E17F65.2040009@gmail.com> On 23/1/2014 10:54 ??, Steven D'Aprano wrote: > On Thu, Jan 23, 2014 at 01:27:50PM +0000, Oscar Benjamin wrote: > >> Steven wrote: >>> With a UTF-8 implementation, won't that mean that string indexing >>> operations are O(N) rather than O(1)? E.g. how do you know which UTF-8 >>> byte(s) to look at to get the character at index 42 without having to >>> walk the string from the start? >> >> With an indexing cache you do this by looking it up in the cache. [...] >> So there's a O(N) per-string cost if the string is >> indexed at least once and an O(k) cost per attempt to index into the string. > > Plus a memory cost for the cache, and increased complexity for indexing > operations. > > >> OTOH using utf-8 internally means that encoding and decoding as utf-8 becomes >> O(1) instead of O(N) and uses less memory since if you do s=b.decode('utf-8') >> then s and b can share the same memory buffer. > > CPython caches the UTF-8 encoding of (some? all?) strings too. It's not > clear to me if they are cached lazily or eagerly, but either way, I'm > not convinced by this argument that this is useful. > > In most applications, you'll decode from UTF-8 exactly once, when you > read the string in (from a file, or the network, say), and you'll encode > it at most once, when you write it out again. Many strings won't ever be > encoded to UTF-8. I have difficulty imagining any application which will > need to repeatedly encode the same string to UTF-8, where the cost > of encoding is more significant than the cost of I/O. One possible > exception might be printing to the terminal, say at the REPL, but even > then, I don't imagine the time savings are worthwhile. I haven't > noticed the REPL in Python 3.3 being particularly more responsive than > that in 3.2. If there's a speed-up, it's invisible to the user, but at > the cost of using extra memory for the UTF-8 cache. > > >> The FSR does this for latin-1 >> strings but utf-8 strings are more useful. So if you're more likely to >> decode/encode from/to utf-8 than you are to index into a long string that's a >> big saving. > > But I don't believe that is the case for any realistic use-case. Just > about every string operation involves walking the string, which > needs to walk the string in characters (code points), not UTF-8 bytes. > At the Python layer, string indexing might be relatively uncommon, but > at the implementation layer, I think every string operation relies on > it. > > But having said all this, I know that using UTF-8 internally for strings > is quite common (e.g. Haskell does it, without even an index cache, and > documents that indexing operations can be slow). CPython's FSR has > received much (in my opinion, entirely uninformed) criticism from one > vocal person in particular for not using UTF-8 internally. If PyPy goes > ahead with using UTF-8 internally, I look forward to comparing memory > and time benchmarks of string operations between CPython and PyPy. > I have to admit that due to my work (databases and data processing), i'm biased towards I/O (UTF-8 is better due to size) rather than CPU. At least from my use cases, the most frequent operations that i do on strings are read, write, store, use them as keys in dicts, concatenate and split. For most of above things (with the exception of split maybe?), an index cache would not be needed, and UTF-8 due to its smaller size would be faster than wide unicode encodings. l. From njh at njhurst.com Fri Jan 24 00:06:48 2014 From: njh at njhurst.com (Nathan Hurst) Date: Fri, 24 Jan 2014 10:06:48 +1100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: <52E17F65.2040009@gmail.com> References: <20140123132746.GA3553@gmail.com> <20140123205414.GL3915@ando> <52E17F65.2040009@gmail.com> Message-ID: <20140123230648.GA9615@ajhurst.org> On Thu, Jan 23, 2014 at 10:45:25PM +0200, Elefterios Stamatogiannakis wrote: > >But having said all this, I know that using UTF-8 internally for strings > >is quite common (e.g. Haskell does it, without even an index cache, and > >documents that indexing operations can be slow). CPython's FSR has > >received much (in my opinion, entirely uninformed) criticism from one > >vocal person in particular for not using UTF-8 internally. If PyPy goes > >ahead with using UTF-8 internally, I look forward to comparing memory > >and time benchmarks of string operations between CPython and PyPy. > > > > I have to admit that due to my work (databases and data processing), > i'm biased towards I/O (UTF-8 is better due to size) rather than > CPU. > > At least from my use cases, the most frequent operations that i do > on strings are read, write, store, use them as keys in dicts, > concatenate and split. > > For most of above things (with the exception of split maybe?), an > index cache would not be needed, and UTF-8 due to its smaller size > would be faster than wide unicode encodings. I hear Steven's points, but my experience matches Elefterios' - smaller data is faster[1]. I'll also note that although many string processing algorithms can be written in terms of indexing, many(most?) are actually stream processing algorithms which do not actually need efficient character offset to/from byte offset calculations. For example, split works by walking the entire string in a single pass outputing substrings as it goes. regards, njh [1] which suggests that lz77ing longer strings by default is not a terrible idea. From oscar.j.benjamin at gmail.com Fri Jan 24 02:25:12 2014 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Fri, 24 Jan 2014 01:25:12 +0000 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: <20140123205414.GL3915@ando> References: <20140123132746.GA3553@gmail.com> <20140123205414.GL3915@ando> Message-ID: On 23 January 2014 20:54, Steven D'Aprano wrote: > On Thu, Jan 23, 2014 at 01:27:50PM +0000, Oscar Benjamin wrote: > >> Steven wrote: >> > With a UTF-8 implementation, won't that mean that string indexing >> > operations are O(N) rather than O(1)? E.g. how do you know which UTF-8 >> > byte(s) to look at to get the character at index 42 without having to >> > walk the string from the start? >> >> With an indexing cache you do this by looking it up in the cache. [...] >> So there's a O(N) per-string cost if the string is >> indexed at least once and an O(k) cost per attempt to index into the string. > > Plus a memory cost for the cache, and increased complexity for indexing > operations. There is an O(n/k) memory cost for the cache, yes and indexing has an O(k) CPU cost. >> OTOH using utf-8 internally means that encoding and decoding as utf-8 becomes >> O(1) instead of O(N) and uses less memory since if you do s=b.decode('utf-8') >> then s and b can share the same memory buffer. > > CPython caches the UTF-8 encoding of (some? all?) strings too. It's not > clear to me if they are cached lazily or eagerly, but either way, I'm > not convinced by this argument that this is useful. There's a difference between caching the utf-8 string and reusing it. If you cache the utf-8 string you're holding two memory buffers: one in FSR format and one in utf-8 format. If your implementation uses utf-8 then you can use one buffer for both the str and the corresponding bytes object. > In most applications, you'll decode from UTF-8 exactly once, when you > read the string in (from a file, or the network, say), and you'll encode > it at most once, when you write it out again. Many strings won't ever be > encoded to UTF-8. That's true but actually the concept of encoding/decoding is a conceptual error at the implementation level. Really all you ever do is re-encode. So why not choose the most common input/output format as the standard internal format if it saves conversion time? > I have difficulty imagining any application which will > need to repeatedly encode the same string to UTF-8, where the cost > of encoding is more significant than the cost of I/O. It's not about repeatedly encoding the same string. It's about the fact that you always need to encode or decode the string at least once. If that's the case then minimising the cost of that one guaranteed encode/decode operation is a good thing. > >> The FSR does this for latin-1 >> strings but utf-8 strings are more useful. So if you're more likely to >> decode/encode from/to utf-8 than you are to index into a long string that's a >> big saving. > > But I don't believe that is the case for any realistic use-case. Just > about every string operation involves walking the string, which > needs to walk the string in characters (code points), not UTF-8 bytes. > At the Python layer, string indexing might be relatively uncommon, but > at the implementation layer, I think every string operation relies on > it. Okay so here's a review of the Python str methods broken down into relevant categories. The first is all the methods that can operate on utf-8 strings by treating them as arbitrary byte strings. For example concatenating two utf-8 strings is equivalent to concatenating the corresponding abstract unicode character strings. I make the assumption here that the number of characters (not bytes) in the string is known (e.g. for ljust): join, ljust, partition, replace, rpartition, rjust, count, endswith, expandtabs, format, format_map, startswith, zfill, __add__, __contains__, __eq__, __ge__, __gt__, __hash__, __le__, __len__, __lt__, __mod__, __mul__, __ne__, __rmul__, __rmod__ The second group is those methods that must understand that the string is encoded in utf-8 but can walk through without needing to jump to an index in the middle of the string: isalnum, isalpha, isdecimal, isdigit, isidentifier, islower, isnumeric, isprintable, isspace, istitle, isupper, lstrip, lower, maketrans, casefold, center, encode, rsplit, rstrip, split, splitlines, strip, swapcase, title, translate, upper, __format__, __iter__ Finally here are the methods that need to be able to jump to an arbitrary character index within the string: find, index, rfind, rindex, __getitem__ In my own use I think that I would most commonly use (in)equality, concatenation, interpolation and formatting rather than indexing or slicing (__getitem__). None of these operation requires using the index cache. > But having said all this, I know that using UTF-8 internally for strings > is quite common (e.g. Haskell does it, without even an index cache, and > documents that indexing operations can be slow). I find that Haskell is generally a bit slow with strings, utf-8 or otherwise. Haskell naturally lends itself to stream type processing so it's understandable that they would see efficient indexing as less important. Using linked-lists for strings is probably never going to be as efficient as well-implemented atomic string types. > CPython's FSR has > received much (in my opinion, entirely uninformed) criticism from one > vocal person in particular for not using UTF-8 internally. I feel like you're putting a jinx on this list. Please don't encourage the "vocal person" to come here! Oscar From chrish at pianocktail.org Fri Jan 24 03:40:00 2014 From: chrish at pianocktail.org (Christian Hudon) Date: Thu, 23 Jan 2014 21:40:00 -0500 Subject: [pypy-dev] Questions on stdlib-2.7.5 failures Message-ID: <52E1D280.3040902@pianocktail.org> Hello, I'm working again on the stdlib-2.7.5 branch of Pypy. I have a couple questions, but since nobody seems to be around on irc right now, I'm asking them here. :-) First, I kicked the buildbots yesterday, and there are a *lot* of errors due to tests not being able to import the MAXREPEAT constant from the _sre module. But it seems to me that it is defined in pypy/module/_sre/__init__.py... Am I missing something? Are the errors due to a build problem or something else? Second, I'm trying to reproduce some of the more interesting failures (test_multiprocessing, test_str), but when I try to run the tests on my laptop, they fail like this instead: platform darwin -- Python 2.7.5 -- pytest-2.2.4.dev2 -- /usr/bin/python collected 1 items test_str.py [snip] Traceback (application-level): File "/users/chrish/Code/pypy/pypy/tool/pytest/run-script/regrverbose.py", line 4 in from test import test_support File "/users/chrish/Code/pypy/lib-python/2.7/test/test_support.py", line 21 in import struct File "/users/chrish/Code/pypy/lib-python/2.7/struct.py", line 1 in from _struct import * ImportError: No module named _struct FAILED I don't really see anything that could look like a module called "_struct" in the source code... Am I missing something here too? Thanks in advance, Christian From benjamin at python.org Fri Jan 24 04:00:03 2014 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 23 Jan 2014 19:00:03 -0800 Subject: [pypy-dev] Questions on stdlib-2.7.5 failures In-Reply-To: <52E1D280.3040902@pianocktail.org> References: <52E1D280.3040902@pianocktail.org> Message-ID: <1390532403.23713.74676069.7A76E5A3@webmail.messagingengine.com> On Thu, Jan 23, 2014, at 06:40 PM, Christian Hudon wrote: > Hello, > > I'm working again on the stdlib-2.7.5 branch of Pypy. I have a couple > questions, but since nobody seems to be around on irc right now, I'm > asking them here. :-) Cool. Did you see lib-python/stdlib-upgrade.txt? > > First, I kicked the buildbots yesterday, and there are a *lot* of errors > due to tests not being able to import the MAXREPEAT constant from the > _sre module. But it seems to me that it is defined in > pypy/module/_sre/__init__.py... Am I missing something? Are the errors > due to a build problem or something else? Where definition of that constant is changed in CPython 2.7.5. You'll probably have to move it accordingly. > > Second, I'm trying to reproduce some of the more interesting failures > (test_multiprocessing, test_str), but when I try to run the tests on my > laptop, they fail like this instead: > > platform darwin -- Python 2.7.5 -- pytest-2.2.4.dev2 -- /usr/bin/python > collected 1 items > test_str.py > [snip] > Traceback (application-level): > File > "/users/chrish/Code/pypy/pypy/tool/pytest/run-script/regrverbose.py", > line 4 in > from test import test_support > File "/users/chrish/Code/pypy/lib-python/2.7/test/test_support.py", > line 21 in > import struct > File "/users/chrish/Code/pypy/lib-python/2.7/struct.py", line 1 in > > from _struct import * > ImportError: No module named _struct > FAILED > > I don't really see anything that could look like a module called > "_struct" in the source code... Am I missing something here too? I suspect the CPython struct.py stub is shadowing pypy's builtin struct module. Try deleting struct.py. From arigo at tunes.org Fri Jan 24 08:54:56 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 08:54:56 +0100 Subject: [pypy-dev] Questions on stdlib-2.7.5 failures In-Reply-To: <52E1D280.3040902@pianocktail.org> References: <52E1D280.3040902@pianocktail.org> Message-ID: Hi Christian, On Fri, Jan 24, 2014 at 3:40 AM, Christian Hudon wrote: > First, I kicked the buildbots yesterday, and there are a *lot* of errors due > to tests not being able to import the MAXREPEAT constant from the _sre > module. But it seems to me that it is defined in > pypy/module/_sre/__init__.py... Am I missing something? Are the errors due > to a build problem or something else? This is merely a detail for a deeper an issue. CPython's "re" format changed between 2.7.3 and 2.7.5 (a potential compatibility issue). We need to do something about it, but as far as I know, what exactly to do is still not clear. > Second, I'm trying to reproduce some of the more interesting failures > (test_multiprocessing, test_str), but when I try to run the tests on my > laptop, they fail like this instead: You are running "py.test test_str.py" on CPython. This is not really fully supported any more. The buildbot runs them as "pypy test_str.py", with the just-translated PyPy. The best way is to extract a part of the failing test and copy the failing few lines only into pypy/objspace/std/test/test_stringobject.py, for example, where it can be run with py.test on CPython and fixed accordingly. A bient?t, Armin. From arigo at tunes.org Fri Jan 24 09:41:11 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 09:41:11 +0100 Subject: [pypy-dev] Questions on stdlib-2.7.5 failures In-Reply-To: References: <52E1D280.3040902@pianocktail.org> Message-ID: Hi Christian, On Fri, Jan 24, 2014 at 8:54 AM, Armin Rigo wrote: >> pypy/module/_sre/__init__.py... Am I missing something? Are the errors due >> to a build problem or something else? > > This is merely a detail for a deeper an issue. CPython's "re" format > changed between 2.7.3 and 2.7.5 (a potential compatibility issue). We > need to do something about it, but as far as I know, what exactly to > do is still not clear. I forgot that I did something on trunk some time ago about it. So I finished fixing the 2.7.5 branch; now the "own" tests at least pass. I didn't try to translate and run the CPython tests, but the MAXREPEAT name issue should not be a problem there. A bient?t, Armin. From arigo at tunes.org Fri Jan 24 11:39:08 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 11:39:08 +0100 Subject: [pypy-dev] PyPy 2 unicode class In-Reply-To: References: <20140123132746.GA3553@gmail.com> <20140123205414.GL3915@ando> Message-ID: Hi all, Thanks everybody for your comments on this topic. Our initial motivation for doing that is to simplify RPython by getting rid of the RPython unicode type. I think that the outcome of these mails is that there is no single obvious answer as to whether the change would benefit or hurt Python applications. The benefits in reduced memory usage might in some applications win or loose to the higher cost of indexing. I personally guess that most applications would benefit marginally, with a few special cases that loose importantly. This would not be a very nice outcome... The design of the index cache is crucial to minimize that... On the other hand, maybe some applications doing I/O on utf-8 strings might benefit a lot. Once we have a good design, the only thing we can do is try it, and measure the results on a range of applications. A bient?t, Armin. From arigo at tunes.org Fri Jan 24 14:15:27 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 14:15:27 +0100 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi, On Thu, Jan 23, 2014 at 10:36 PM, wrote: >> I was considering the possibility of taking an index or id() of the object >> and just casting it into "void*" form, and then converting it back and >> using it to look up objects in a list/dict somewhere.. > > Alex Pyattaev did this (is in the pypy-dev mail archives, somewhere around > early Summer 2012). This exists in CFFI (see ffi.new_handle()). More generally, wouldn't it make some sense to try to bring cppyy closer to cffi? It is a change of some of the basics, so only suitable for people without an existing code base. What I'm thinking of is if we could consider a cppyy-like interface as an extension of the cffi interface, like C++ is an extension of C, or if it doesn't really make sense. A bient?t, Armin. From arigo at tunes.org Fri Jan 24 14:25:36 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 14:25:36 +0100 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: Hi, On Thu, Jan 23, 2014 at 5:28 PM, Maciej Fijalkowski wrote: > Since I have absolutely no idea what you're talking about, let's start with: Same for me. Also, using another SSL library like PyOpenSSL is a good idea anyway --- on both CPython and PyPy. If I had a choice I'd prefer to scrap the _ssl module from PyPy, but we can't because CPython has it too. I wouldn't mind if using _ssl produced warnings on PyPy. The alternative, which is coming up with better security features in the _ssl module, seems like a lot of work, of questionable use and safety (just use PyOpenSSL), and bound to break CPython compatibility. A bient?t, Armin. From arigo at tunes.org Fri Jan 24 14:33:04 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 14:33:04 +0100 Subject: [pypy-dev] PyD and PyPy [was Boost.Python and PyPy -- status?] In-Reply-To: <1390152490.5003.10.camel@anglides.winder.org.uk> References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> <1390152490.5003.10.camel@anglides.winder.org.uk> Message-ID: Hi Russel, On Sun, Jan 19, 2014 at 6:28 PM, Russel Winder wrote: > I am delving into PyD just now to get it working with CPython 3 as well > as CPython 2. Can you point me towards the best way of trying to make > this also work with PyPy and PyPy3? Sorry, you're not getting answers because your question is too vague. What is PyD, what kind of C or C++ code it is, ...? If it's C, we recommend CFFI. If it's C++, we don't have a one-size-fits-all story for that --- at all. If it's the kind of C++ code that can easily be embedded into a C interface, we generally recommend that you do that and then use CFFI as well. A bient?t, Armin. From piotr.skamruk at gmail.com Fri Jan 24 15:11:10 2014 From: piotr.skamruk at gmail.com (Piotr Skamruk) Date: Fri, 24 Jan 2014 15:11:10 +0100 Subject: [pypy-dev] PyD and PyPy [was Boost.Python and PyPy -- status?] In-Reply-To: References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> <1390152490.5003.10.camel@anglides.winder.org.uk> Message-ID: 2014/1/24 Armin Rigo : > [...] > Sorry, you're not getting answers because your question is too vague. > What is PyD, what kind of C or C++ code it is, ...? If it's C, we > recommend CFFI. If it's C++, we don't have a one-size-fits-all story > for that --- at all. If it's the kind of C++ code that can easily be > embedded into a C interface, we generally recommend that you do that > and then use CFFI as well. Probably http://pyd.dsource.org/ is this about what You are asking. It's wrapper of python api for http://dlang.org/ From chef at ghum.de Fri Jan 24 14:51:43 2014 From: chef at ghum.de (Massa, Harald Armin) Date: Fri, 24 Jan 2014 14:51:43 +0100 Subject: [pypy-dev] PyD and PyPy [was Boost.Python and PyPy -- status?] In-Reply-To: References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> <1390152490.5003.10.camel@anglides.winder.org.uk> Message-ID: 2014/1/24 Armin Rigo : > What is PyD PYD (Put You Down) is a song by Justin Bieber, a singer especially popular with girls in the age bracket 10-14. https://www.youtube.com/watch?v=0Os-eKGQzpw Its lyrics use many words to describe the protagonist encouraging a lady to have sexual intercourse with him. Personally I doubt that those lyrics are appropriate for the afformentioned age bracket, but I shall not judge other cultures. Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 From arigo at tunes.org Fri Jan 24 18:13:50 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 18:13:50 +0100 Subject: [pypy-dev] PyD and PyPy [was Boost.Python and PyPy -- status?] In-Reply-To: References: <8738kxt7jk.fsf@tsmithe.net> <1389289516.2811.137.camel@newpride> <87ha9drpx4.fsf@tsmithe.net> <1390152490.5003.10.camel@anglides.winder.org.uk> Message-ID: Hi Piotr, On Fri, Jan 24, 2014 at 3:11 PM, Piotr Skamruk wrote: > Probably http://pyd.dsource.org/ is this about what You are asking. > It's wrapper of python api for http://dlang.org/ I actually guessed that much from googling. What I'm asking is the kind of C++ code that it needs to interact with, which is not obvious just by staring at that page. Given time, I know I could probably explore pyd.dsource.org and dlang.org and find out myself, but I'm trying here to explain to the original author of this thread why his question remained unanswered --- because nobody here happens to be familiar with : there are just too many such projects out there. A bient?t, Armin. From wlavrijsen at lbl.gov Fri Jan 24 18:28:42 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 24 Jan 2014 09:28:42 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Armin, > More generally, wouldn't it make some sense to try to bring cppyy closer > to cffi? yes, it does. Is high on my wish list (and not just the programmer-facing side, also the internals although there are several things that are close but don't quite fit). At the top is cling, though, and time is lacking. Of course, if you know a volunteer who can help here? :) Thanks, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From arigo at tunes.org Fri Jan 24 18:37:02 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 18:37:02 +0100 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Wim, On Fri, Jan 24, 2014 at 6:28 PM, wrote: > yes, it does. Is high on my wish list (and not just the programmer-facing > side, also the internals although there are several things that are close > but don't quite fit). I meant specifically the way the interface is to be used by the end programmer, ignoring its implementation for now. This would mean moving to more cffi-like idioms: removing the implicit ownership-tracking logic, not guessing too hard about which overloaded function is meant to be called, and so on --- generally "explicit is better than implicit" and if the end programmer needs to expose some library in a simpler interface, he can use a Python layer on top of it (or write his own). So this would end up somewhere slightly different than where cppyy is currently. A bient?t, Armin. From wlavrijsen at lbl.gov Fri Jan 24 18:56:39 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 24 Jan 2014 09:56:39 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Armin, > I meant specifically the way the interface is to be used by the end > programmer, ignoring its implementation for now. This would mean > moving to more cffi-like idioms: removing the implicit > ownership-tracking logic, not guessing too hard about which overloaded > function is meant to be called, and so on that is all possible, and largely there, just the other way around than: > --- generally "explicit is > better than implicit" and if the end programmer needs to expose some > library in a simpler interface, he can use a Python layer on top of it > (or write his own). So this would end up somewhere slightly different > than where cppyy is currently. By default, there is memory tracking, auto-casting, overloading, template instantiation (with cling; partially with cint), etc. And if that is not desired, a Python layer can be written to do things differently. E.g. to select a specific overload, use the "__dispatch__" function. I don't think that that will be workable, though. Maybe I should have bitten on the "like C++ is an extension of C" comment. :) Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From arigo at tunes.org Fri Jan 24 21:41:07 2014 From: arigo at tunes.org (Armin Rigo) Date: Fri, 24 Jan 2014 21:41:07 +0100 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Wim, On Fri, Jan 24, 2014 at 6:56 PM, wrote: > By default, there is memory tracking, auto-casting, overloading, template > instantiation (with cling; partially with cint), etc. And if that is not > desired, a Python layer can be written to do things differently. E.g. to > select a specific overload, use the "__dispatch__" function. So you're basically answering "no" :-) As you're describing it, cppyy is at the "ctypes" level of providing automatic everything and lots of options and ways to work around them if they are not wanted. CFFI is at the opposite end of the spectrum on this respect. But that is for C. I don't have enough practical knowledge of C++ to judge how viable a "CPPFFI" would be. A bient?t, Armin. From wlavrijsen at lbl.gov Fri Jan 24 22:35:11 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Fri, 24 Jan 2014 13:35:11 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Armin, > So you're basically answering "no" :-) no, I'm not saying no. (Okay, now I did twice. :) ) I did say that it was high on my wish list, after all. I know that there are many people who like to work the explicit way, which is why such an interface needs to be provided. And it can be done, as it already (mostly) exists, just scattered. Further, module level stuff already lives on 'cppyy.' and C++ stuff lives on 'cppyy.gbl'. So I think that this can be totally supported and faithfully follow CFFI, without clashing. I just think that it's a historic artefact that folks want this, not that they have good reasons for it. CFFI has an additional advantage, by not having heavy dependencies. Good luck writing your own pycppparser, though. :/ (Separate from ABI concerns.) So if with C++, a full C++ compiler is pulled into the mix anyway, why content with so much hand-coding? > As you're describing it, cppyy is at the "ctypes" level of providing > automatic everything There's nothing automatic about ctypes. :? What I mean is something like e.g. std::map. Why would anyone want to re-invent the pythonizations, if one can just do: for key, value in my_cpp_map: # ... whatever directly from the automatic bindings and pythonizations provided by cppyy? > and lots of options and ways to work around them if they are not wanted Right, but it's work either way? If no automation is provided, then that work that could have been automated, needs to be provided by hand. To me the equation seems to be that I rather have the automation get it 95% right, with 5% (possibly frustrating) fixup, than doing 100% by hand even if that gives me (non-frustrating) full control. I know that some people do not agree with that, which is why I DO want to have a CPPFFI, so that I can simply point them to that and let them have at it. It's their own feet, after all. :) > I don't have enough practical knowledge of C++ to judge how viable a > "CPPFFI" would be. C++11 is the big deal. I'm not saying that all of a sudden folks are going to write better quality code (not to mention the installed base), but the standards committee seems to have finally gotten it in their heads that if intentions are not specified in the interface, it creates problems for humans. I expect a lot of cleanup there, so that automation can grow from 95% right to 99% right or so, becoming a total no-brainer. For example, with move semantics, returning an std::vector from a function is almost as cheap as returning a pointer to an array, without any of the questions about object ownership. (And don't fall over that "almost": most likely the extra 8 bytes of the size data member that are copied over in the case of the vector sit on the same cache line anyway, with the highest cost being the pointer indirection in both cases.) Of course, on top of all that, there will always be folks who think that such automation produces something that "feels" too much like C++ and too little like python. And they're highly likely to be right. But designing such a python-like API is arguably more pleasant on top of C++-like python than on C++ directly. Anyway. :) As long as it is clear I didn't say "no", but said "after the cling backend is in." (And yes, given the delays, I know that reads a bit like "when pigs fly," but that's really not how I intend it.) Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From foogod at gmail.com Fri Jan 24 23:32:00 2014 From: foogod at gmail.com (Alex Stewart) Date: Fri, 24 Jan 2014 14:32:00 -0800 Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: On Fri, Jan 24, 2014 at 12:41 PM, Armin Rigo wrote: > On Fri, Jan 24, 2014 at 6:56 PM, wrote: > > By default, there is memory tracking, auto-casting, overloading, template > > instantiation (with cling; partially with cint), etc. And if that is not > > desired, a Python layer can be written to do things differently. E.g. to > > select a specific overload, use the "__dispatch__" function. > > So you're basically answering "no" :-) As you're describing it, cppyy > is at the "ctypes" level of providing automatic everything and lots of > options and ways to work around them if they are not wanted. CFFI is > at the opposite end of the spectrum on this respect. But that is for > C. I don't have enough practical knowledge of C++ to judge how viable > a "CPPFFI" would be. > Speaking as somebody who is currently using cppyy to build a Python binding for a large already-existing third-party C++ library, I have to say I think that moving cppyy in that direction would be a substantial step *backwards*, making cppyy substantially less valuable and useful. Please don't do this. The CFFI way of approaching things works well for C, because C was already designed as a pretty explicit language to begin with. The problem, however, is large portions of the C++ language itself are designed with the assumption that lots of implicit stuff will be done behind the scenes by the compiler to make it something a human can actually use. If you take that away, C++ becomes much, much more cumbersome and onerous (verging on completely unusable), and I believe any Python binding that takes that away would also be similarly cumbersome. The end result would be something where everyone would *have* to code a (fairly complex) extra Python interface layer for every single class they wanted to expose, just to make it usable in Python in a way that approximates how it's intended to be used in C++ (which as a secondary issue would probably make it all slower, to boot). In my opinion, the thing that currently makes cppyy one of the best cross-language-binding implementations I've ever seen is that it almost completely eliminates the need to do any of that. I was able to make a working binding (that works almost exactly the same way the C++ docs say it should) for somewhere over 90% of this roughly 150,000 line C++ library just by writing up a short 30-line XML file and running it through a couple of utilities to build it.. it's only the last 5-10% or so I'm having to work on now by hand and write extra interface logic for (and much of that promises to be fixed with cling). That is, frankly, amazingly valuable to me. In general, I'm a big fan of "explicit is better than implicit", but I do not believe that that always translates to cross-language bindings. The entire point of good language bindings is to do as much of the translation work automatically as possible so you don't have to write it all yourself every single time. If the language you're translating to is designed to do lots of things implicitly, it's really a practical necessity that the binding do the same, or it's really only doing half its job. (I agree that there are a few areas where cppyy could be better at *allowing* you to be explicit if you want to, but that does not sound like what you were proposing.. Apologies if I misunderstood anything.) --Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.rade at gmail.com Sun Jan 26 13:10:01 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Sun, 26 Jan 2014 13:10:01 +0100 Subject: [pypy-dev] Translating PyPy on Windows Message-ID: Hi everyone! When I translated PyPy on Windows, on a laptop with an Intel 3210M processor, I got the following timings: annotate --- 464.5 s rtype_lltype --- 882.7 s pyjitpl_lltype --- 825.9 s backendopt_lltype --- 259.2 s stackcheckinsertion_lltype --- 309.0 s database_c --- 349.6 s source_c --- 502.4 s compile_c --- 3830.5 s =========================================== Total: --- 7423.8 s More than half the time is spent compiling C code. The makefile uses neither precompiled headers nor parallel compilation, so there is lots of room for improvement. I tried these two optimizations, and here are the timings for the C code compilation: baseline: 3830.5 s (as above) pc: 1301.2 s pch: 404.8 s pch + pc: 165.8 s pc = parallel compilation pch = precompiled headers Using both optimizations you get a factor 23 speed up! Adding parallel compilation is easy. Adding precompiled headers is a bit more work; here I did a quick fix that is not suitable for production code, but good enough for profiling. I plan to add parallel compilation on Windows, along with some other Windows related fixes, to the PyPy code. I will not add precompiled headers now. Armin, Maciej: Does that sound OK? ---------------------------------------------------------- The main gain when using precompiled headers comes from the header forwarddecl.h. This header is 17.8 MB and is included in 635 source files. The amount of C code that is compiled is about 424 MB. But because of forwarddecl.h, you are in fact compiling another 635 x 17.8 MB = 11 GB. Compiling 424 MB or 11 GB of C code - that does make a difference. But you can not just create a precompiled header that includes forwarddecl.h and apply it to all source files. You should only apply the precompiled header to files that include forwarddecl.h to begin with. These seem to be precisely the source files with names that match one of the patterns data_*.c, nonfuncnodes*.c, implement*.c, pypy_*.c and rpython_*.c. But I don't see how to write a nmake rule that only applies to such files. (nmake does not have as good pattern matching support as GNU make.) --Johan From fijall at gmail.com Sun Jan 26 13:34:53 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 26 Jan 2014 13:34:53 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: Message-ID: On Sun, Jan 26, 2014 at 1:10 PM, Johan R?de wrote: > Hi everyone! > > When I translated PyPy on Windows, > on a laptop with an Intel 3210M processor, > I got the following timings: > > annotate --- 464.5 s > rtype_lltype --- 882.7 s > pyjitpl_lltype --- 825.9 s > backendopt_lltype --- 259.2 s > stackcheckinsertion_lltype --- 309.0 s > database_c --- 349.6 s > source_c --- 502.4 s > compile_c --- 3830.5 s > =========================================== > Total: --- 7423.8 s > > More than half the time is spent compiling C code. > The makefile uses neither precompiled headers nor parallel compilation, > so there is lots of room for improvement. > I tried these two optimizations, > and here are the timings for the C code compilation: > > baseline: 3830.5 s (as above) > pc: 1301.2 s > pch: 404.8 s > pch + pc: 165.8 s > > pc = parallel compilation > pch = precompiled headers > > Using both optimizations you get a factor 23 speed up! > > Adding parallel compilation is easy. > > Adding precompiled headers is a bit more work; > here I did a quick fix that is not suitable for production code, > but good enough for profiling. > > I plan to add parallel compilation on Windows, > along with some other Windows related fixes, to the PyPy code. > I will not add precompiled headers now. > > Armin, Maciej: Does that sound OK? > > ---------------------------------------------------------- > > The main gain when using precompiled headers comes from the header > forwarddecl.h. > This header is 17.8 MB and is included in 635 source files. > The amount of C code that is compiled is about 424 MB. > But because of forwarddecl.h, you are in fact compiling another 635 x 17.8 > MB = 11 GB. > Compiling 424 MB or 11 GB of C code - that does make a difference. > > But you can not just create a precompiled header that includes forwarddecl.h > and apply it to all source files. > You should only apply the precompiled header to files that include > forwarddecl.h to begin with. > These seem to be precisely the source files with names that match one of the > patterns > data_*.c, nonfuncnodes*.c, implement*.c, pypy_*.c and rpython_*.c. > But I don't see how to write a nmake rule that only applies to such files. > (nmake does not have as good pattern matching support as GNU make.) > > --Johan > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev Hi Johan Should this mail come with some attachments? Cheers, fijal From johan.rade at gmail.com Sun Jan 26 13:44:11 2014 From: johan.rade at gmail.com (=?UTF-8?B?Sm9oYW4gUsOlZGU=?=) Date: Sun, 26 Jan 2014 13:44:11 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: Message-ID: <52E5031B.6050903@gmail.com> On 2014-01-26 13:34, Maciej Fijalkowski wrote: > > Hi Johan > > Should this mail come with some attachments? > > Cheers, > fijal > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > Hi Maciej, No attachments. I haven't written the actual Python code yet, I just hacked the generated makefile by hand. Cheers, Johan From johan.rade at gmail.com Sun Jan 26 13:44:11 2014 From: johan.rade at gmail.com (=?UTF-8?B?Sm9oYW4gUsOlZGU=?=) Date: Sun, 26 Jan 2014 13:44:11 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: Message-ID: <52E5031B.6050903@gmail.com> On 2014-01-26 13:34, Maciej Fijalkowski wrote: > > Hi Johan > > Should this mail come with some attachments? > > Cheers, > fijal > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > Hi Maciej, No attachments. I haven't written the actual Python code yet, I just hacked the generated makefile by hand. Cheers, Johan From matti.picus at gmail.com Sun Jan 26 15:15:33 2014 From: matti.picus at gmail.com (matti picus) Date: Sun, 26 Jan 2014 16:15:33 +0200 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: <52E5031B.6050903@gmail.com> References: <52E5031B.6050903@gmail.com> Message-ID: On Sun, Jan 26, 2014 at 2:44 PM, Johan R?de wrote: > On 2014-01-26 13:34, Maciej Fijalkowski wrote: > > >> Hi Johan >> >> Should this mail come with some attachments? >> >> Cheers, >> fijal >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > Hi Maciej, > > No attachments. > I haven't written the actual Python code yet, > I just hacked the generated makefile by hand. > > Cheers, > > Johan > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > If you wish, post the hacked Makefile to a pastebin and we can work backwards from there. I would be willing to help move this forward. Matti -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.rade at gmail.com Sun Jan 26 16:25:31 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Sun, 26 Jan 2014 16:25:31 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: Message-ID: <52E528EB.4090907@gmail.com> Hi Maciej, Here are the details of what I did when I tried using precompiled headers. 1. I checked out revision 68823 2. I translated PyPy. 3. I went to the directory with all (well most) of the sources. On my machine it was C:\Users\Rade\AppData\Local\Temp\usession-default-0\testing_1 4. I added include guards to common_header.h and preimpl.h 5. I added a file stdafx.c with the single line #include "stdafx.h" 6. I added a file stdafx.h with the the lines #ifndef PYPY_STDAFX_H #define PYPY_STDAFX_H #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #endif 7. I revised the makefile so that it would created a precompiled header from stdafx.c and compile all the other source files with that precompiled header. The commands were essentially cl.exe stdafx.c $(CFLAGS) /Ycstdafx.h /Fpstdafx.pch $(INCLUDEDIRS) cl.exe $(CFLAGS) /Yustdafx.h /Fpstdafx.pch /FIstdafx.h $(INCLUDEDIRS) Here /Yc = create precompiled header /Yu = use precompiled header /Fp = specify precompiled header name /FI = force inclusion of header file Things would of course be different with gcc. Then I ran the makefile. Several files did not compile. 8. I revised the makefile so that the source files that did not compile in step 7 were compiled without the precompiled header. I ran the makefile again. It compiled and linked and the generated executable seemed to work. ---------------------------------- What should be done in addition: 1. The precompiled header should only be used with files that begin with #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" That seems to be the files with names that match one of the patterns data_*.h nonfuncnodes*.h implement*.h pypy_*.h rpython_*.h The remaining files either include python.h or or just plain standard C lib headers. 2. The source files typically begin with something like #define PYPY_FILE_NAME "rpython_rtyper_lltypesystem_rlist.c" #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #include "src/g_include.h" The first line is different in each file. This seems to indicate that you can not use precompiled headers. However, the macro PYPY_FILE_NAME is only used in src/g_include.h. Thus the files could, and should, be rewritten as #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #define PYPY_FILE_NAME "rpython_rtyper_lltypesystem_rlist.c" #include "src/g_include.h" Now it becomes clear that you can put the first four lines in a precompiled header. 3. Optionally one could refactor src/g_include.h so it does not rely on the macro PYPY_FILE_NAME. Then you could add src/g_include.h to the precompiled header as well. But I think the performance gain from that would be small. Cheers, Johan From johan.rade at gmail.com Sun Jan 26 16:33:33 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Sun, 26 Jan 2014 16:33:33 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: <52E5031B.6050903@gmail.com> Message-ID: On 2014-01-26 15:15, matti picus wrote: > On Sun, Jan 26, 2014 at 2:44 PM, Johan R?de > > If you wish, post the hacked Makefile to a pastebin and we can work > backwards from there. > I would be willing to help move this forward. > Matti Hi Matti, Here it is https://dl.dropboxusercontent.com/u/525329/Makefile It is the first makefile I write ever. You have been warned. --Johan From johan.rade at gmail.com Sun Jan 26 16:25:31 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Sun, 26 Jan 2014 16:25:31 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: Message-ID: <52E528EB.4090907@gmail.com> Hi Maciej, Here are the details of what I did when I tried using precompiled headers. 1. I checked out revision 68823 2. I translated PyPy. 3. I went to the directory with all (well most) of the sources. On my machine it was C:\Users\Rade\AppData\Local\Temp\usession-default-0\testing_1 4. I added include guards to common_header.h and preimpl.h 5. I added a file stdafx.c with the single line #include "stdafx.h" 6. I added a file stdafx.h with the the lines #ifndef PYPY_STDAFX_H #define PYPY_STDAFX_H #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #endif 7. I revised the makefile so that it would created a precompiled header from stdafx.c and compile all the other source files with that precompiled header. The commands were essentially cl.exe stdafx.c $(CFLAGS) /Ycstdafx.h /Fpstdafx.pch $(INCLUDEDIRS) cl.exe $(CFLAGS) /Yustdafx.h /Fpstdafx.pch /FIstdafx.h $(INCLUDEDIRS) Here /Yc = create precompiled header /Yu = use precompiled header /Fp = specify precompiled header name /FI = force inclusion of header file Things would of course be different with gcc. Then I ran the makefile. Several files did not compile. 8. I revised the makefile so that the source files that did not compile in step 7 were compiled without the precompiled header. I ran the makefile again. It compiled and linked and the generated executable seemed to work. ---------------------------------- What should be done in addition: 1. The precompiled header should only be used with files that begin with #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" That seems to be the files with names that match one of the patterns data_*.h nonfuncnodes*.h implement*.h pypy_*.h rpython_*.h The remaining files either include python.h or or just plain standard C lib headers. 2. The source files typically begin with something like #define PYPY_FILE_NAME "rpython_rtyper_lltypesystem_rlist.c" #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #include "src/g_include.h" The first line is different in each file. This seems to indicate that you can not use precompiled headers. However, the macro PYPY_FILE_NAME is only used in src/g_include.h. Thus the files could, and should, be rewritten as #include "common_header.h" #include "structdef.h" #include "forwarddecl.h" #include "preimpl.h" #define PYPY_FILE_NAME "rpython_rtyper_lltypesystem_rlist.c" #include "src/g_include.h" Now it becomes clear that you can put the first four lines in a precompiled header. 3. Optionally one could refactor src/g_include.h so it does not rely on the macro PYPY_FILE_NAME. Then you could add src/g_include.h to the precompiled header as well. But I think the performance gain from that would be small. Cheers, Johan From donald at stufft.io Mon Jan 27 00:07:37 2014 From: donald at stufft.io (Donald Stufft) Date: Sun, 26 Jan 2014 18:07:37 -0500 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: On Jan 24, 2014, at 8:25 AM, Armin Rigo wrote: > Hi, > > On Thu, Jan 23, 2014 at 5:28 PM, Maciej Fijalkowski wrote: >> Since I have absolutely no idea what you're talking about, let's start with: > > Same for me. Also, using another SSL library like PyOpenSSL is a good > idea anyway --- on both CPython and PyPy. > > If I had a choice I'd prefer to scrap the _ssl module from PyPy, but > we can't because CPython has it too. I wouldn't mind if using _ssl > produced warnings on PyPy. The alternative, which is coming up with > better security features in the _ssl module, seems like a lot of work, > of questionable use and safety (just use PyOpenSSL), and bound to > break CPython compatibility. > > > A bient?t, > > Armin. Sorry for the delay in responding to this. Currently what I would like to do is backport the SSL hostname verification from CPython 3.x to PyPy. Currently on CPython 2.x and PyPy if you wish to verify the hostname of SSL connections you need to manually copy this backport into your code and use that. I think that this is a sad state of affairs and would prefer to have it back ported to both CPython and PyPy. The code I?ve seen in the wild tends to check for the existence of the functions that enable this and fall back if they don?t exist. So this should just silently work for people. The benefit of doing it in PyPy instead of each individual project is that it gives PyPy users a central location to update if changes are needed to be made instead of needing to do this in each project that does it. The compatibly implications of this are: 1) Software written for CPython will continue to work out of the box with PyPy 2) Someone could write software that only worked on PyPy and CPython 3.x but not CPython 2.x The CPython developers have thus far decided that back porting hostname verification to CPython 2.x is not worth breaking the policy of no new features in 2.7. I disagree with this conclusion fwiw. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From roberto at unbit.it Mon Jan 27 07:06:49 2014 From: roberto at unbit.it (Roberto De Ioris) Date: Mon, 27 Jan 2014 07:06:49 +0100 Subject: [pypy-dev] Building -pie binary Message-ID: Hi, while trying to simplify the usage of pypy in uWSGI, i started working with pie mode (gcc -pie option). Position independent executables can be dl-opened (at least) on Linux and OSX as normal shared libraries. As dealing with libpypy-c is basically messy, this could be a solution. I think distributing a -pie enabled binary package would push adoption of uWSGI+pypy solutions (currently the biggest deployment i have found rely on libpypy-c distributed by my company that is months old), and generally will simplify the embedding of pypy in other apps. I still have not tried to translate a pypy source tree with -pie flag, so maybe nothing will work (even if i do not see why). I will make an attempt in the next few hours Regards -- Roberto De Ioris http://unbit.it From fijall at gmail.com Mon Jan 27 08:38:52 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 27 Jan 2014 08:38:52 +0100 Subject: [pypy-dev] Building -pie binary In-Reply-To: References: Message-ID: Hi Roberto. I'll look into it (I know I wanted to do it for a few months now though ;-) Cheers, fijal On Mon, Jan 27, 2014 at 7:06 AM, Roberto De Ioris wrote: > > Hi, while trying to simplify the usage of pypy in uWSGI, i started working > with pie mode (gcc -pie option). > > Position independent executables can be dl-opened (at least) on Linux and > OSX as normal shared libraries. > > As dealing with libpypy-c is basically messy, this could be a solution. > > I think distributing a -pie enabled binary package would push adoption of > uWSGI+pypy solutions (currently the biggest deployment i have found rely > on libpypy-c distributed by my company that is months old), and generally > will simplify the embedding of pypy in other apps. > > I still have not tried to translate a pypy source tree with -pie flag, so > maybe nothing will work (even if i do not see why). I will make an attempt > in the next few hours > > Regards > > -- > Roberto De Ioris > http://unbit.it > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Mon Jan 27 12:13:37 2014 From: arigo at tunes.org (Armin Rigo) Date: Mon, 27 Jan 2014 12:13:37 +0100 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: Hi Donald, On 27 January 2014 00:07, Donald Stufft wrote: > The CPython developers have thus far decided that back porting hostname verification to > CPython 2.x is not worth breaking the policy of no new features in 2.7. I disagree with this > conclusion fwiw. You're considering adding a feature to PyPy 2 that is not present in CPython 2.7. Our general policy is that this is a bad idea. How about you create and distribute a regular CPython C extension module? If you do, and (for example) it is written with CFFI, then it will also work out of the box on PyPy. (I also remind you that you have not answered the question "why not use PyOpenSSL instead".) A bient?t, Armin. From donald at stufft.io Mon Jan 27 13:09:43 2014 From: donald at stufft.io (Donald Stufft) Date: Mon, 27 Jan 2014 07:09:43 -0500 Subject: [pypy-dev] Validate SSL/TLS In-Reply-To: References: <29C41FFD-89A9-4B3B-9678-114E608420DF@stufft.io> Message-ID: <0BA7268D-DBC1-4764-8B75-621B29DE6A28@stufft.io> On Jan 27, 2014, at 6:13 AM, Armin Rigo wrote: > Hi Donald, > > On 27 January 2014 00:07, Donald Stufft wrote: >> The CPython developers have thus far decided that back porting hostname verification to >> CPython 2.x is not worth breaking the policy of no new features in 2.7. I disagree with this >> conclusion fwiw. > > You're considering adding a feature to PyPy 2 that is not present in > CPython 2.7. Our general policy is that this is a bad idea. Ok, Well IDK the policies I was complaining about CPython not doing this and Alex told me to raise the issue on the PyPy mailing list :) > How > about you create and distribute a regular CPython C extension module? > If you do, and (for example) it is written with CFFI, then it will > also work out of the box on PyPy. It doesn?t need to be C, it?d be implemented in pure Python. > > (I also remind you that you have not answered the question "why not > use PyOpenSSL instead?.) PyOpenSSL (nor does OpenSSL) does not contain host name validation code. OpenSSL has historically said that this is up to the app to provide. > > > A bient?t, > > Armin. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP using GPGMail URL: From lamalaxmi7 at gmail.com Tue Jan 28 17:53:56 2014 From: lamalaxmi7 at gmail.com (Little Girlie) Date: Tue, 28 Jan 2014 11:53:56 -0500 Subject: [pypy-dev] Your message on craigslist Message-ID: <86734B86.3AEB0305@gmail.com> An HTML attachment was scrubbed... URL: From wlavrijsen at lbl.gov Wed Jan 29 02:31:07 2014 From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov) Date: Tue, 28 Jan 2014 17:31:07 -0800 (PST) Subject: [pypy-dev] cppyy and callbacks In-Reply-To: References: Message-ID: Hi Alex, On Fri, 24 Jan 2014, Alex Stewart wrote: > Speaking as somebody who is currently using cppyy to build a Python binding > for a large already-existing third-party C++ library, I have to say I think > that moving cppyy in that direction would be a substantial step *backwards*, > making cppyy substantially less valuable and useful. Please don't do this. no, that's not the plan. The two approaches can live next to each other. > I agree that there are a few areas where cppyy could be better at > *allowing* you to be explicit if you want to Right, but I've never found an occasion where a C++ helper did not resolve such problems. And with Cling, these helpers can be embedded (CFFI-style), so that may be enough of a solution. (Dunno, is a matter of try-and-see.) Btw., just to show I have some code (and to allow anyone to comment if there are any takers), I've checked in some code that is the first step towards a Cling backend. Is still far from functional, but I have to start somewhere. Best regards, Wim -- WLavrijsen at lbl.gov -- +1 (510) 486 6411 -- www.lavrijsen.net From ladyo54 at yahoo.com Wed Jan 29 06:40:30 2014 From: ladyo54 at yahoo.com (Little Girlie) Date: Wed, 29 Jan 2014 00:40:30 -0500 Subject: [pypy-dev] Gday Mate! Message-ID: An HTML attachment was scrubbed... URL: From ron.reiter at gmail.com Thu Jan 30 09:23:01 2014 From: ron.reiter at gmail.com (Ron Reiter) Date: Thu, 30 Jan 2014 10:23:01 +0200 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: I've discovered the cause of the leak - Cyclone uses BytesIO instead of StringIO. Turns out the following code leaks on Pypy only! from io import BytesIO while True: a = BytesIO() I'm filing a bug right now. On Thu, Jan 23, 2014 at 6:27 PM, Maciej Fijalkowski wrote: > On Thu, Jan 23, 2014 at 2:44 PM, Marko Tasic wrote: > >> Hi Ron, >> >> Exactly same problem I have discovered yesterday in one of my production >> servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and tornado==3.2 >> on ArchLinux updated on daily basis. I do not have more details on this, >> but I can say only that POST requests takes at least 10x slower then on >> CPython. However, I measured time spent inside method and its 0.01s, and >> actual sending of response takes 30s. This is very strange for reason. >> >> How I can help on this and send you more details? >> >> Regards, >> Marko Tasic >> > > gunicorn is a bit dodge (it's pypy support is not very official), but I > can give it a go - please write some sort of reproducible example and we > can take it from there. > > >> >> >> On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: >> >>> Something really strange is happening to me. I've stumbled upon a memory >>> leak that happens only when answering a POST request using Twisted and Pypy. >>> >>> The problem is described here: >>> >>> >>> http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak >>> >>> It could either be a bug in the Python code that handles the request and >>> leaks, a but in Twisted, or in Pypy. >>> >>> I've been investing the problem for weeks now, with the following >>> conclusion: >>> >>> * CPython does not leak with this code >>> * Tornado with Pypy does not leak with this code >>> * Answering GET requests instead of POST requests does not leak with >>> this code >>> * I tried switching GCs, collecting using gc.collect and setting >>> PYPY_GC_MAX, those attempts did not help >>> * This happens both on Linux and Mac >>> >>> Insights as to how to debug this are very welcome. >>> >>> Thanks, >>> >>> -- >>> >>> Ron Reiter >>> [image: Facebook] [image: Twitter] [image: >>> LinkedIn] >>> Contact me: [image: Skype] ron.reiter >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> https://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > -- Ron Reiter [image: Facebook] [image: Twitter] [image: LinkedIn] Contact me: [image: Skype] ron.reiter -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 30 11:19:08 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 30 Jan 2014 11:19:08 +0100 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: I can definitely reproduce it, investigating On Thu, Jan 30, 2014 at 9:23 AM, Ron Reiter wrote: > I've discovered the cause of the leak - Cyclone uses BytesIO instead of > StringIO. > Turns out the following code leaks on Pypy only! > > from io import BytesIO > while True: a = BytesIO() > > I'm filing a bug right now. > > > On Thu, Jan 23, 2014 at 6:27 PM, Maciej Fijalkowski wrote: > >> On Thu, Jan 23, 2014 at 2:44 PM, Marko Tasic wrote: >> >>> Hi Ron, >>> >>> Exactly same problem I have discovered yesterday in one of my production >>> servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and tornado==3.2 >>> on ArchLinux updated on daily basis. I do not have more details on this, >>> but I can say only that POST requests takes at least 10x slower then on >>> CPython. However, I measured time spent inside method and its 0.01s, and >>> actual sending of response takes 30s. This is very strange for reason. >>> >>> How I can help on this and send you more details? >>> >>> Regards, >>> Marko Tasic >>> >> >> gunicorn is a bit dodge (it's pypy support is not very official), but I >> can give it a go - please write some sort of reproducible example and we >> can take it from there. >> >> >>> >>> >>> On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: >>> >>>> Something really strange is happening to me. I've stumbled upon a >>>> memory leak that happens only when answering a POST request using Twisted >>>> and Pypy. >>>> >>>> The problem is described here: >>>> >>>> >>>> http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak >>>> >>>> It could either be a bug in the Python code that handles the request >>>> and leaks, a but in Twisted, or in Pypy. >>>> >>>> I've been investing the problem for weeks now, with the following >>>> conclusion: >>>> >>>> * CPython does not leak with this code >>>> * Tornado with Pypy does not leak with this code >>>> * Answering GET requests instead of POST requests does not leak with >>>> this code >>>> * I tried switching GCs, collecting using gc.collect and setting >>>> PYPY_GC_MAX, those attempts did not help >>>> * This happens both on Linux and Mac >>>> >>>> Insights as to how to debug this are very welcome. >>>> >>>> Thanks, >>>> >>>> -- >>>> >>>> Ron Reiter >>>> [image: Facebook] [image: Twitter] [image: >>>> LinkedIn] >>>> Contact me: [image: Skype] ron.reiter >>>> >>>> _______________________________________________ >>>> pypy-dev mailing list >>>> pypy-dev at python.org >>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>> >>>> >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> https://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> > > > -- > > Ron Reiter > [image: Facebook] [image: Twitter] [image: > LinkedIn] > Contact me: [image: Skype] ron.reiter > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 30 11:38:42 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 30 Jan 2014 11:38:42 +0100 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: Should be fixed by 40fa4f3a0740, try tomorrow. On Thu, Jan 30, 2014 at 11:19 AM, Maciej Fijalkowski wrote: > I can definitely reproduce it, investigating > > > On Thu, Jan 30, 2014 at 9:23 AM, Ron Reiter wrote: > >> I've discovered the cause of the leak - Cyclone uses BytesIO instead of >> StringIO. >> Turns out the following code leaks on Pypy only! >> >> from io import BytesIO >> while True: a = BytesIO() >> >> I'm filing a bug right now. >> >> >> On Thu, Jan 23, 2014 at 6:27 PM, Maciej Fijalkowski wrote: >> >>> On Thu, Jan 23, 2014 at 2:44 PM, Marko Tasic wrote: >>> >>>> Hi Ron, >>>> >>>> Exactly same problem I have discovered yesterday in one of my >>>> production servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and >>>> tornado==3.2 on ArchLinux updated on daily basis. I do not have more >>>> details on this, but I can say only that POST requests takes at least 10x >>>> slower then on CPython. However, I measured time spent inside method and >>>> its 0.01s, and actual sending of response takes 30s. This is very strange >>>> for reason. >>>> >>>> How I can help on this and send you more details? >>>> >>>> Regards, >>>> Marko Tasic >>>> >>> >>> gunicorn is a bit dodge (it's pypy support is not very official), but I >>> can give it a go - please write some sort of reproducible example and we >>> can take it from there. >>> >>> >>>> >>>> >>>> On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: >>>> >>>>> Something really strange is happening to me. I've stumbled upon a >>>>> memory leak that happens only when answering a POST request using Twisted >>>>> and Pypy. >>>>> >>>>> The problem is described here: >>>>> >>>>> >>>>> http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak >>>>> >>>>> It could either be a bug in the Python code that handles the request >>>>> and leaks, a but in Twisted, or in Pypy. >>>>> >>>>> I've been investing the problem for weeks now, with the following >>>>> conclusion: >>>>> >>>>> * CPython does not leak with this code >>>>> * Tornado with Pypy does not leak with this code >>>>> * Answering GET requests instead of POST requests does not leak with >>>>> this code >>>>> * I tried switching GCs, collecting using gc.collect and setting >>>>> PYPY_GC_MAX, those attempts did not help >>>>> * This happens both on Linux and Mac >>>>> >>>>> Insights as to how to debug this are very welcome. >>>>> >>>>> Thanks, >>>>> >>>>> -- >>>>> >>>>> Ron Reiter >>>>> [image: Facebook] [image: >>>>> Twitter] [image: LinkedIn] >>>>> Contact me: [image: Skype] ron.reiter >>>>> >>>>> _______________________________________________ >>>>> pypy-dev mailing list >>>>> pypy-dev at python.org >>>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> pypy-dev mailing list >>>> pypy-dev at python.org >>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>> >>>> >>> >> >> >> -- >> >> Ron Reiter >> [image: Facebook] [image: Twitter] [image: >> LinkedIn] >> Contact me: [image: Skype] ron.reiter >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ron.reiter at gmail.com Thu Jan 30 11:51:17 2014 From: ron.reiter at gmail.com (Ron Reiter) Date: Thu, 30 Jan 2014 12:51:17 +0200 Subject: [pypy-dev] Memory leaks in Pypy In-Reply-To: References: Message-ID: Thanks! On Jan 30, 2014 12:39 PM, "Maciej Fijalkowski" wrote: > Should be fixed by 40fa4f3a0740, try tomorrow. > > > On Thu, Jan 30, 2014 at 11:19 AM, Maciej Fijalkowski wrote: > >> I can definitely reproduce it, investigating >> >> >> On Thu, Jan 30, 2014 at 9:23 AM, Ron Reiter wrote: >> >>> I've discovered the cause of the leak - Cyclone uses BytesIO instead of >>> StringIO. >>> Turns out the following code leaks on Pypy only! >>> >>> from io import BytesIO >>> while True: a = BytesIO() >>> >>> I'm filing a bug right now. >>> >>> >>> On Thu, Jan 23, 2014 at 6:27 PM, Maciej Fijalkowski wrote: >>> >>>> On Thu, Jan 23, 2014 at 2:44 PM, Marko Tasic wrote: >>>> >>>>> Hi Ron, >>>>> >>>>> Exactly same problem I have discovered yesterday in one of my >>>>> production servers running PyPy 2.2.1, Flask==0.10.1, gunicorn==18.0 and >>>>> tornado==3.2 on ArchLinux updated on daily basis. I do not have more >>>>> details on this, but I can say only that POST requests takes at least 10x >>>>> slower then on CPython. However, I measured time spent inside method and >>>>> its 0.01s, and actual sending of response takes 30s. This is very strange >>>>> for reason. >>>>> >>>>> How I can help on this and send you more details? >>>>> >>>>> Regards, >>>>> Marko Tasic >>>>> >>>> >>>> gunicorn is a bit dodge (it's pypy support is not very official), but I >>>> can give it a go - please write some sort of reproducible example and we >>>> can take it from there. >>>> >>>> >>>>> >>>>> >>>>> On Wed, Jan 22, 2014 at 2:08 PM, Ron Reiter wrote: >>>>> >>>>>> Something really strange is happening to me. I've stumbled upon a >>>>>> memory leak that happens only when answering a POST request using Twisted >>>>>> and Pypy. >>>>>> >>>>>> The problem is described here: >>>>>> >>>>>> >>>>>> http://stackoverflow.com/questions/21063842/using-twisted-cyclone-pypy-to-handle-post-requests-cause-a-memory-leak >>>>>> >>>>>> It could either be a bug in the Python code that handles the request >>>>>> and leaks, a but in Twisted, or in Pypy. >>>>>> >>>>>> I've been investing the problem for weeks now, with the following >>>>>> conclusion: >>>>>> >>>>>> * CPython does not leak with this code >>>>>> * Tornado with Pypy does not leak with this code >>>>>> * Answering GET requests instead of POST requests does not leak with >>>>>> this code >>>>>> * I tried switching GCs, collecting using gc.collect and setting >>>>>> PYPY_GC_MAX, those attempts did not help >>>>>> * This happens both on Linux and Mac >>>>>> >>>>>> Insights as to how to debug this are very welcome. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> -- >>>>>> >>>>>> Ron Reiter >>>>>> [image: Facebook] [image: >>>>>> Twitter] [image: LinkedIn] >>>>>> Contact me: [image: Skype] ron.reiter >>>>>> >>>>>> _______________________________________________ >>>>>> pypy-dev mailing list >>>>>> pypy-dev at python.org >>>>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>>>> >>>>>> >>>>> >>>>> _______________________________________________ >>>>> pypy-dev mailing list >>>>> pypy-dev at python.org >>>>> https://mail.python.org/mailman/listinfo/pypy-dev >>>>> >>>>> >>>> >>> >>> >>> -- >>> >>> Ron Reiter >>> [image: Facebook] [image: Twitter] [image: >>> LinkedIn] >>> Contact me: [image: Skype] ron.reiter >>> >> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuaxo2 at yahoo.com Thu Jan 30 12:56:24 2014 From: stuaxo2 at yahoo.com (Stuart Axon) Date: Thu, 30 Jan 2014 03:56:24 -0800 (PST) Subject: [pypy-dev] Chocolatey package for windows ? Message-ID: <1391082984.62885.YahooMailNeo@web140902.mail.bf1.yahoo.com> http://chocolatey.org/ bills itself as apt-get for windows, it's fairly convenient - any chance of adding a pypy package ? If so, then one could install with?cinst pypy S++ -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 30 13:28:06 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 30 Jan 2014 13:28:06 +0100 Subject: [pypy-dev] Chocolatey package for windows ? In-Reply-To: <1391082984.62885.YahooMailNeo@web140902.mail.bf1.yahoo.com> References: <1391082984.62885.YahooMailNeo@web140902.mail.bf1.yahoo.com> Message-ID: On Thu, Jan 30, 2014 at 12:56 PM, Stuart Axon wrote: > http://chocolatey.org/ bills itself as apt-get for windows, it's fairly > convenient - any chance of adding a pypy package ? > > If so, then one could install with cinst pypy > > S++ I don't think it's a question for the core developers. Who is maintaining those packages? If you're asking whether you can do it, then the answer is obviously yes, there are no restrictions how you can use or distribute pypy. Cheers, fijal From rymg19 at gmail.com Thu Jan 30 17:56:27 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Thu, 30 Jan 2014 10:56:27 -0600 Subject: [pypy-dev] Chocolatey package for windows ? In-Reply-To: <1391082984.62885.YahooMailNeo@web140902.mail.bf1.yahoo.com> References: <1391082984.62885.YahooMailNeo@web140902.mail.bf1.yahoo.com> Message-ID: I'll do that, cause it sounds great. However, it'll take about 1 week. I'm pretty busy right now. On Thu, Jan 30, 2014 at 5:56 AM, Stuart Axon wrote: > http://chocolatey.org/ bills itself as apt-get for windows, it's fairly > convenient - any chance of adding a pypy package ? > > If so, then one could install with cinst pypy > > S++ > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From stuaxo2 at yahoo.com Thu Jan 30 18:19:23 2014 From: stuaxo2 at yahoo.com (Stuart Axon) Date: Thu, 30 Jan 2014 09:19:23 -0800 (PST) Subject: [pypy-dev] Chocolatey package for windows ? In-Reply-To: References: <1391082984.62885.YahooMailNeo@web140902.mail.bf1.yahoo.com> Message-ID: <1391102363.42514.YahooMailNeo@web140906.mail.bf1.yahoo.com> Cheers, I was reticent to volunteer as there's quite a lot on, I'll certainly test it though. ? S++ On Thursday, January 30, 2014 5:03 PM, Ryan Gonzalez wrote: I'll do that, cause it sounds great. However, it'll take about 1 week. I'm pretty busy right now. > > > >On Thu, Jan 30, 2014 at 5:56 AM, Stuart Axon wrote: > >http://chocolatey.org/ bills itself as apt-get for windows, it's fairly convenient - any chance of adding a pypy package ? >> >> >> >>If so, then one could install with?cinst pypy >> >> >>S++ >>_______________________________________________ >>pypy-dev mailing list >>pypy-dev at python.org >>https://mail.python.org/mailman/listinfo/pypy-dev >> >> > > > >-- > >Ryan >If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From pfoster at goldengate.org Fri Jan 31 02:58:38 2014 From: pfoster at goldengate.org (Yvette Cross) Date: Fri, 31 Jan 2014 02:58:38 +0100 Subject: [pypy-dev] All orders are shipped via Express Courier delivery to ensure fast and prompt delivery. Message-ID: <672383689.40568957714397@goldengate.org> Hello Pypy You can prefer any watch for her that matches her taste and your resources. The perfect place to buy watches as we offer best quality, excellent service, money back guarantee in case you are not satisfied and have many other strong points, such as fast delivery, helpful and caring customer service. ******************************************* About two weeks ago, I ordered two watches. A Rolex Daytona, and a Vacheron Constantin. I found your web site by chance, and decided to take a gamble and order these two watches. To my surprise and delight, my watches arrived in just under a week. The watches are amazing. We checked the Daytona against a real one, and you cant tell the difference. First class all the way. I just received my third watch, an A. Lange & Sohne!!!! The watch is fantastic!!! Thanks! Yvette Cross ******************************************* Click here ---> http://watchpros.ru