From sasharaud at gmail.com Thu Jan 3 08:30:56 2013 From: sasharaud at gmail.com (=?KOI8-R?B?4czFy9PBzsTSIPLB1cQ=?=) Date: Thu, 3 Jan 2013 11:30:56 +0400 Subject: [pypy-dev] translating own language to shared lib Message-ID: Hi! It's my very first question posted to mailing lists so please correct me if I am doing something wrong. I have subscribed to python mailing lists with araud at mail.ru address (which is mirrored to gmail) but I am writing from sasharaud at gmail.com - I don't know if it matters or not. Anyways, question is next: Is there anywhere a minimal working example of an own language translated to shared library, not standalone executable? (I saw standalone example and it worked for me, now I need more :) ) What I need is to make a problem-specific extension written in RPython that is densely integrated with exiting C++ code (quite a big product). Requirements are next: the result has to be a library (.dll and .so: product is cross-platform) with C interface (not executable). Library will be loaded by C++ code and will be called as if it was written in C: calls always come from product threads (but during calls the library can call given callbacks). I know that this is possible because cpyext does it. But cpyext is too big to understand minimal required set of actions for: * declaring target as shared lib, not standalone * running translation with proper arguments (seems like "-shared" is not enough as I get this: [translation:WARNING] target specific arguments supplied but will be ignored: -shared) * declaration of API (I saw decorators but I didn't get if decorator is enough or anything else like function registration is required) If there's no ready example I can create and publish example based on my investigation. But I still need someone's help to answer questions above. Thanks in advance! With respect, Alexander. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri Jan 4 19:08:45 2013 From: arigo at tunes.org (Armin Rigo) Date: Fri, 4 Jan 2013 19:08:45 +0100 Subject: [pypy-dev] translating own language to shared lib In-Reply-To: References: Message-ID: Hi Araud, On Thu, Jan 3, 2013 at 8:30 AM, ????????? ???? wrote: > Is there anywhere a minimal working example of an own language translated to > shared library, not standalone executable? As we discussed on irc: you need "translate.py --shared targetfoobar.py". This makes a .dll and a .exe. The interface of the .dll is very basic: it has just one "entry point" function that is called by the .exe. I doubt it is enough for your purpose, where you want two-way communications between the "libfoobar.dll" and your C++ program. This is not supported so far. I could imagine to split this function into a few, like "initialize yourself" and "run". You could pass extra arguments to the "initialize" function, like a pointer to a table of function pointers, which the dll stores somewhere and uses to call back C++. This is all work that you would need to do yourself, though. (Finally, the issue of Win64 not being supported for now: it's not going to magically solve itself. Someone needs to step forward and fix it, which is work too.) A bient?t, Armin. From arigo at tunes.org Fri Jan 4 19:40:34 2013 From: arigo at tunes.org (Armin Rigo) Date: Fri, 4 Jan 2013 19:40:34 +0100 Subject: [pypy-dev] translating own language to shared lib In-Reply-To: References: Message-ID: Hi, Regarding your other question: "how to embed PyPy in C/C++ application", missing _Py_Initialize. It's not supported for now but could be, given enough work. Here is an alternative, assuming that you don't need to have an interface compatible with CPython (the PyXxx functions). We could come up with some cffi-based solution (http://cffi.readthedocs.org/). For example, with minimal amount of work, we could give you the possibility to do something like this: * The C/C++ code loads "libpypy-c.dll" and calls some function 'initialize("initial_script.py")'. (Or, running on CPython, it would load "libpython27.dll" and call _PyInitialize(); PyRun_File("initial_script.py");.) * initial_script.py uses CFFI in a slightly different way than documented above (the other way around, basically): it declares a C interface, but with C functions that end up calling back into Python. (These C functions themselves are not meant to be called *from* Python.) * Then initial_script.py finishes and returns the name of the library built by "ffi.verify()". * The C/C++ side then loads and uses that library --- a regular-looking C library with whatever custom C API that you declared above. The trick is that this lets you write a regular library that can be called from C, with the C API you choose, but *implemented as Python code*. I imagine that this could be generally interesting. A bient?t, Armin. From matti.picus at gmail.com Sat Jan 5 23:04:05 2013 From: matti.picus at gmail.com (Matti Picus) Date: Sun, 06 Jan 2013 00:04:05 +0200 Subject: [pypy-dev] document builds on readthedocs have been failing for quite a while Message-ID: <50E8A355.3050409@gmail.com> Looking at https://readthedocs.org/builds/pypy shows that builds of pypy documentation have been failing Actually, the failures are not recent, as far back as Dec 1 the builds were failing, when I checked just now this page shows successful builds https://readthedocs.org/builds/pypy/?page=46 while here they began to fail https://readthedocs.org/builds/pypy/?page=45 (your mileage may vary, I think the page numbers change over time) The sphinx failure seems to say something about vanilla (yum!), ------------------------------------------------------------- Exception occurred: File "/var/build/user_builds/pypy/checkouts/latest/pypy/config/makerestdoc.py", line 217, in config_role opt = getattr(h._cfgimpl_descr, n) AttributeError: 'OptionDescription' object has no attribute 'vanilla' -------------------------------------------------------------- I found a changeset in pypy that modified the --vanilla option suspiciously around the time the failures began : -------------------------------- 53b204769d36 remove vanilla option, which is totally unused --------------------------------- Also, "make html" in the doc directory fails for me (win32, default branch), however fixing that failure did not make sphinx happier on readthedocs, see pypy changeset b36997015a40 and the resulting document build on readthedocs https://readthedocs.org/builds/pypy/386842/ I give up, perhaps someone else can fix this. Matti From matti.picus at gmail.com Mon Jan 7 09:24:21 2013 From: matti.picus at gmail.com (matti picus) Date: Mon, 7 Jan 2013 10:24:21 +0200 Subject: [pypy-dev] windows build crashing - command line too long Message-ID: The number of c files in a pypy translation increased by 5, which pushed the length of the link command line over the windows limit. The solution is to use a response file instead of the command line. I will fix this over the next while unless someone else does it first, but until then expect complaints that windows translation is broken. Here is the error message: NMAKE : fatal error U1095: expanded command line ... too long From fijall at gmail.com Mon Jan 7 11:31:45 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 7 Jan 2013 12:31:45 +0200 Subject: [pypy-dev] windows build crashing - command line too long In-Reply-To: References: Message-ID: On Mon, Jan 7, 2013 at 10:24 AM, matti picus wrote: > The number of c files in a pypy translation increased by 5, which > pushed the length of the link command line over the windows limit. > The solution is to use a response file instead of the command line. > I will fix this over the next while unless someone else does it first, > but until then expect complaints that windows translation is broken. > Here is the error message: > NMAKE : fatal error U1095: expanded command line ... too long May I say "what the fuck"? Although I think unix tools also have limit (4096 was one for rm a while ago) From matti.picus at gmail.com Mon Jan 7 15:29:32 2013 From: matti.picus at gmail.com (matti picus) Date: Mon, 7 Jan 2013 16:29:32 +0200 Subject: [pypy-dev] document builds on readthedocs have been failing for quite a while In-Reply-To: <50E8A355.3050409@gmail.com> References: <50E8A355.3050409@gmail.com> Message-ID: it seems the problem is in pypy/doc/config/generate.py which recreates (copies?) *.rst from *.txt for all the command line options but does not remove _unused_ old versions of the rst files. When we removed a command line option, the corresponding txt file was removed from hg, but the rst file was not removed. Then sphinx picks up all the rst files and boom. Anyone want to suggest a fix? Anyone know why we do txt -> rst in the first place? Matti On Sun, Jan 6, 2013 at 12:04 AM, Matti Picus wrote: > Looking at https://readthedocs.org/builds/pypy shows that builds of pypy > documentation have been failing > Actually, the failures are not recent, as far back as Dec 1 the builds were > failing, when I checked just now this page shows successful builds > https://readthedocs.org/builds/pypy/?page=46 > while here they began to fail > https://readthedocs.org/builds/pypy/?page=45 > (your mileage may vary, I think the page numbers change over time) > > The sphinx failure seems to say something about vanilla (yum!), > ------------------------------------------------------------- > Exception occurred: > File > "/var/build/user_builds/pypy/checkouts/latest/pypy/config/makerestdoc.py", > line 217, in config_role > opt = getattr(h._cfgimpl_descr, n) > AttributeError: 'OptionDescription' object has no attribute 'vanilla' > -------------------------------------------------------------- > I found a changeset in pypy that modified the --vanilla option suspiciously > around the time the failures began : > -------------------------------- > 53b204769d36 remove vanilla option, which is totally unused > --------------------------------- > Also, "make html" in the doc directory fails for me (win32, default branch), > however fixing that failure did not make sphinx happier on readthedocs, > see pypy changeset b36997015a40 > and the resulting document build on readthedocs > https://readthedocs.org/builds/pypy/386842/ > > I give up, perhaps someone else can fix this. > Matti From fijall at gmail.com Mon Jan 7 16:35:10 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 7 Jan 2013 17:35:10 +0200 Subject: [pypy-dev] document builds on readthedocs have been failing for quite a while In-Reply-To: References: <50E8A355.3050409@gmail.com> Message-ID: On Mon, Jan 7, 2013 at 4:29 PM, matti picus wrote: > it seems the problem is in pypy/doc/config/generate.py which recreates > (copies?) *.rst from *.txt for all the command line options > but does not remove _unused_ old versions of the rst files. When we > removed a command line option, the corresponding txt file > was removed from hg, but the rst file was not removed. Then sphinx > picks up all the rst files and boom. > Anyone want to suggest a fix? > Anyone know why we do txt -> rst in the first place? > Matti > uh. ask people on #rtd maybe? > On Sun, Jan 6, 2013 at 12:04 AM, Matti Picus wrote: >> Looking at https://readthedocs.org/builds/pypy shows that builds of pypy >> documentation have been failing >> Actually, the failures are not recent, as far back as Dec 1 the builds were >> failing, when I checked just now this page shows successful builds >> https://readthedocs.org/builds/pypy/?page=46 >> while here they began to fail >> https://readthedocs.org/builds/pypy/?page=45 >> (your mileage may vary, I think the page numbers change over time) >> >> The sphinx failure seems to say something about vanilla (yum!), >> ------------------------------------------------------------- >> Exception occurred: >> File >> "/var/build/user_builds/pypy/checkouts/latest/pypy/config/makerestdoc.py", >> line 217, in config_role >> opt = getattr(h._cfgimpl_descr, n) >> AttributeError: 'OptionDescription' object has no attribute 'vanilla' >> -------------------------------------------------------------- >> I found a changeset in pypy that modified the --vanilla option suspiciously >> around the time the failures began : >> -------------------------------- >> 53b204769d36 remove vanilla option, which is totally unused >> --------------------------------- >> Also, "make html" in the doc directory fails for me (win32, default branch), >> however fixing that failure did not make sphinx happier on readthedocs, >> see pypy changeset b36997015a40 >> and the resulting document build on readthedocs >> https://readthedocs.org/builds/pypy/386842/ >> >> I give up, perhaps someone else can fix this. >> Matti > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Mon Jan 7 18:53:51 2013 From: arigo at tunes.org (Armin Rigo) Date: Mon, 7 Jan 2013 18:53:51 +0100 Subject: [pypy-dev] document builds on readthedocs have been failing for quite a while In-Reply-To: References: <50E8A355.3050409@gmail.com> Message-ID: Hi Matti, On Mon, Jan 7, 2013 at 3:29 PM, matti picus wrote: > Anyone know why we do txt -> rst in the first place? Look at any pair .txt/.rst: the .txt is the check-in file, and the .rst is generated to contain automatically-generated cruft. I think that generate.py can be fixed to first delete all .rst files in this directory (they are listed in .hgignore). A bient?t, Armin. From matti.picus at gmail.com Mon Jan 7 19:04:45 2013 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 07 Jan 2013 20:04:45 +0200 Subject: [pypy-dev] document builds on readthedocs have been failing for quite a while In-Reply-To: References: <50E8A355.3050409@gmail.com>

Message-ID: <50EB0E3D.60804@gmail.com> The problem was that we relied on our nonstandard makefile to do stuff, but readthedocs uses a standard one. Fixed in changeset 832cba6f5385, improvements are welcome. Matti On 7/01/2013 7:53 PM, Armin Rigo wrote: > Hi Matti, > > On Mon, Jan 7, 2013 at 3:29 PM, matti picus wrote: >> Anyone know why we do txt -> rst in the first place? > Look at any pair .txt/.rst: the .txt is the check-in file, and the > .rst is generated to contain automatically-generated cruft. I think > that generate.py can be fixed to first delete all .rst files in this > directory (they are listed in .hgignore). > > > A bient?t, > > Armin. From cfbolz at gmx.de Fri Jan 11 11:17:04 2013 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 11 Jan 2013 11:17:04 +0100 Subject: [pypy-dev] [pypy-commit] pypy default: Optimize match.group('name') by making it a module dict. In-Reply-To: <20130105025525.54A641C01EE@cobra.cs.uni-duesseldorf.de> References: <20130105025525.54A641C01EE@cobra.cs.uni-duesseldorf.de> Message-ID: <50EFE6A0.7010202@gmx.de> Hi Alex, I don't really understand the first change of this commit. Why is it a good idea to change the groupdict of the re parser to be a module dict? There are supposed to be "not too many" module dicts, because they are promoted on read. So I don't get why this is a sensible change. Would you please add a comment to the point where the module dict is instantiated why this is a good idea, and ideally also a test_pypy_c test. Cheers, Carl Friedrich On 01/05/2013 03:55 AM, alex_gaynor wrote: > Author: Alex Gaynor > Branch: > Changeset: r59708:3d2ff1e85bf5 > Date: 2013-01-04 18:55 -0800 > http://bitbucket.org/pypy/pypy/changeset/3d2ff1e85bf5/ > > Log: Optimize match.group('name') by making it a module dict. > > diff --git a/lib-python/2.7/sre_parse.py b/lib-python/2.7/sre_parse.py > --- a/lib-python/2.7/sre_parse.py > +++ b/lib-python/2.7/sre_parse.py > @@ -16,6 +16,12 @@ > > from sre_constants import * > > +try: > + from __pypy__ import newdict > +except ImportError:@ > + def newdict(tp): > + return {} > + > SPECIAL_CHARS = ".\\[{()*+?^$|" > REPEAT_CHARS = "*+?{" > > @@ -68,7 +74,7 @@ > self.flags = 0 > self.open = [] > self.groups = 1 > - self.groupdict = {} > + self.groupdict = newdict("module") > def opengroup(self, name=None): > gid = self.groups > self.groups = gid + 1 > diff --git a/pypy/module/_sre/interp_sre.py b/pypy/module/_sre/interp_sre.py > --- a/pypy/module/_sre/interp_sre.py > +++ b/pypy/module/_sre/interp_sre.py > @@ -90,7 +90,7 @@ > # SRE_Pattern class > > class W_SRE_Pattern(Wrappable): > - _immutable_fields_ = ["code", "flags", "num_groups"] > + _immutable_fields_ = ["code", "flags", "num_groups", "w_indexgroup"] > > def cannot_copy_w(self): > space = self.space > _______________________________________________ > pypy-commit mailing list > pypy-commit at python.org > http://mail.python.org/mailman/listinfo/pypy-commit > From Ronny.Pfannschmidt at gmx.de Fri Jan 11 11:56:59 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 11 Jan 2013 11:56:59 +0100 Subject: [pypy-dev] on linking to older glibc ABI's Message-ID: <50EFEFFB.2070003@gmx.de> Hello, since glibc actually carries backward compatible abi versions, i just wondered if it was possible to link to those older abi versions even on new systems in order to support older systems. so i goggled a bit and found an answer on http://stackoverflow.com/questions/4032373/linking-against-an-old-version-of-libc-to-provide-greater-application-coverage it appears that its enough to emit something like __asm__(".symver realpath,realpath at GLIBC_2.2.5"); in order to request the older symbol versions i think if instructions like those where added when writing out the c code, the night-lies could work on other systems like Debian again. i'd like to ask for opinions on adding such statements and a starting point on how to integrate it with the ExternalCompilationInfo. best, Ronny From Ronny.Pfannschmidt at gmx.de Fri Jan 11 13:21:56 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Fri, 11 Jan 2013 13:21:56 +0100 Subject: [pypy-dev] on linking to older glibc ABI's In-Reply-To: <50EFEFFB.2070003@gmx.de> References: <50EFEFFB.2070003@gmx.de> Message-ID: <50F003E4.7010101@gmx.de> on further investigation there are only 2 symbols that mess it up the first one is __fdelt_chk - its used by the _FORTIFY_SOURCE feature of some FD_* macros related to select, requiring 2.15 abi. the second one is requiring memcpy on the 2.14 abi as far as i can tell all other symbols go for the old versions -- Ronny On 01/11/2013 11:56 AM, Ronny Pfannschmidt wrote: > Hello, > > since glibc actually carries backward compatible abi versions, > i just wondered if it was possible to link to those older abi versions > even on new systems in order to support older systems. > > so i goggled a bit and found an answer on > http://stackoverflow.com/questions/4032373/linking-against-an-old-version-of-libc-to-provide-greater-application-coverage > > > it appears that its enough to emit something like > > __asm__(".symver realpath,realpath at GLIBC_2.2.5"); > > in order to request the older symbol versions > i think if instructions like those where added when writing out the c > code, the night-lies could work on other systems like Debian again. > > i'd like to ask for opinions on adding such statements and a starting > point on how to integrate it with the ExternalCompilationInfo. > > best, > Ronny > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From alex.gaynor at gmail.com Fri Jan 11 16:57:11 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 11 Jan 2013 07:57:11 -0800 Subject: [pypy-dev] [pypy-commit] pypy default: Optimize match.group('name') by making it a module dict. In-Reply-To: <50EFE6A0.7010202@gmx.de> References: <20130105025525.54A641C01EE@cobra.cs.uni-duesseldorf.de> <50EFE6A0.7010202@gmx.de> Message-ID: Hi Carl, The reason is that the dict has similar properties to a module dict: 1) keys are written only once 2) lookups are almost always by constant strings In typical usage a the groupindex dict is never mutated after its initial creation, and reads from it are by a precise name of a field, therefore by having it be a moduledict we can make re.group('name') be free. I'll go ahead and add a comment with this info. Alex On Fri, Jan 11, 2013 at 2:17 AM, Carl Friedrich Bolz wrote: > Hi Alex, > > I don't really understand the first change of this commit. Why is it a > good idea to change the groupdict of the re parser to be a module dict? > There are supposed to be "not too many" module dicts, because they are > promoted on read. So I don't get why this is a sensible change. > > Would you please add a comment to the point where the module dict is > instantiated why this is a good idea, and ideally also a test_pypy_c > test. > > Cheers, > > Carl Friedrich > > > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From cfbolz at gmx.de Fri Jan 11 16:59:01 2013 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 11 Jan 2013 16:59:01 +0100 Subject: [pypy-dev] [pypy-commit] pypy default: Optimize match.group('name') by making it a module dict. In-Reply-To: References: <20130105025525.54A641C01EE@cobra.cs.uni-duesseldorf.de> <50EFE6A0.7010202@gmx.de> Message-ID: <50F036C5.2080604@gmx.de> On 01/11/2013 04:57 PM, Alex Gaynor wrote: > Hi Carl, > > The reason is that the dict has similar properties to a module dict: > > 1) keys are written only once > 2) lookups are almost always by constant strings > > In typical usage a the groupindex dict is never mutated after its > initial creation, and reads from it are by a precise name of a field, > therefore by having it be a moduledict we can make re.group('name') be free. > > I'll go ahead and add a comment with this info. > does that mean that the dict (which is created during parsing) is stored on the regex object? If yes, that is the connection that I didn't understand. CF > Alex > > On Fri, Jan 11, 2013 at 2:17 AM, Carl Friedrich Bolz > wrote: > > Hi Alex, > > I don't really understand the first change of this commit. Why is it a > good idea to change the groupdict of the re parser to be a module dict? > There are supposed to be "not too many" module dicts, because they are > promoted on read. So I don't get why this is a sensible change. > > Would you please add a comment to the point where the module dict is > instantiated why this is a good idea, and ideally also a test_pypy_c > test. > > Cheers, > > Carl Friedrich > > > > > > > -- > "I disapprove of what you say, but I will defend to the death your right > to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) > "The people's good is the highest law." -- Cicero From alex.gaynor at gmail.com Fri Jan 11 17:00:35 2013 From: alex.gaynor at gmail.com (Alex Gaynor) Date: Fri, 11 Jan 2013 08:00:35 -0800 Subject: [pypy-dev] [pypy-commit] pypy default: Optimize match.group('name') by making it a module dict. In-Reply-To: <50F036C5.2080604@gmx.de> References: <20130105025525.54A641C01EE@cobra.cs.uni-duesseldorf.de> <50EFE6A0.7010202@gmx.de> <50F036C5.2080604@gmx.de> Message-ID: AH, yes, it is. This dict is used when you do something like: re.match(r'(?P\d+', '12').group('name') Alex On Fri, Jan 11, 2013 at 7:59 AM, Carl Friedrich Bolz wrote: > >> I'll go ahead and add a comment with this info. >> >> > does that mean that the dict (which is created during parsing) is > stored on the regex object? If yes, that is the connection that I > didn't understand. > -- "I disapprove of what you say, but I will defend to the death your right to say it." -- Evelyn Beatrice Hall (summarizing Voltaire) "The people's good is the highest law." -- Cicero -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Wed Jan 16 17:15:06 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 16 Jan 2013 18:15:06 +0200 Subject: [pypy-dev] NumPyPy 2013 Developer Position Message-ID: Introduction ------------ Proposed herein is a part-time fellowship for developing NumPy in PyPy. The work will initially consist of 100 hours with the possibility of extension, until the funds run out. Development and improvement of PyPy's NumPyPy (as with most Open Source and Free Software) is done as a collaborative process between volunteer, paid, and academic contributors. Due to a successful funding drive but a lack of contributors willing to work directly for PyPy, we find ourselves in the enviable situation of being able to offer this position. Background ---------- PyPy's developers make all PyPy software available to the public without charge, under PyPy's Open Source copyright license, the permissive MIT License. PyPy's license assures that PyPy is equally available to everyone freely on terms that allow both non-commercial and commercial activity. This license allows for academics, for-profit software developers, volunteers and enthusiasts alike to collaborate together to make a better Python implementation for everyone. NumPy support for PyPy is licensed similarly, and therefore NumPy in PyPy support can directly help researchers and developers who seek to do numeric computing but want an easier programming language to use than Fortan or C, which is typically used for these applications. Being licensed freely to the general public means that opportunities to use, improve and learn about how NumPy in PyPy works itself will be generally available to everyone. The Need for a Part-Time Developer ----------------------------------- NumPy project in PyPy has seen some slow, but steady progress since we started working about a year ago. On one hand, it's actually impressive what we could deliver with the effort undertaken, on the other hand, we would like to see the development accelerated. PyPy has strict coding, testing, documentation, and review standards, which ensures excellent code quality, continually improving documentation and code test coverage, and minimal regressions. A part-time developer will be able to bring us closer to the goal of full numpy-api implementation and speed improvements. Work Plan --------- The current proposal is split into two parts: * **Compatibility**: This part covers the core NumPy Python API. We'll implement most NumPy APIs that are officially documented and we'll pass most of NumPy's tests that cover documented APIs and are not implementation details. Specifically, we don't plan to: * implement NumPy's C API * implement other scientific libraries, like SciPy, matplotlib or biopython * implement details that are otherwise agreed by consensus to not have a place in PyPy's implementation of NumPy or agreed with NumPy community to be implementation details * **Speed**: This part will cover significant speed improvements in the JIT that would make numeric computations faster. This includes, but is not necesarilly limited to: * write a set of benchmarks covering various use cases * teaching the JIT backend (or multiple backends) how to deal with vector operations, like SSE * experiments with automatic parallelization using multiple threads, akin to numexpr * improving the JIT register allocator that will make a difference, especially for tight loops As with all speed improvements, it's relatively hard to predict exactly how it'll cope, however we expect the results to be withing an order of magnitude of handwritten C equivalent. Position Candidate -------------------- We would like people who are proficient in NumPy and PyPy (but don't have to be core developers of either) to step up. The developer selection will be done by consensus of PyPy core developers and consulted with the Software Freedom Conservancy for lack of conflict of interest. The main criterium will be past contributions to the PyPy project, but they don't have to be significant in size. A candidate for the Developer position will demonstrate the following: * The ability to write clear, stable, suitable and tested code * The ability to understand and extend the JIT capabilities used in NumPyPy. * A positive presence in PyPy's online community on IRC and the mailing list. Ideally the Developer will also: * Have familiarity with the infrastructure of the PyPy project (including bug tracker and buildbot). * Have Worked to provide education or outreach on PyPy in other forums such as workshops, conferences, and user groups. Conservancy and PyPy are excited to announce the Developer Position. Renumeration for the position will be at the rate of 60 USD per hour, through the Software Freedom Conservancy. PyPy community is promising to provide necessary guidance and help into the current codebase, however we expect a successful candidate to be able to review code and incorporate external patches within two months of the starting date of the contract. Candidates should submit their proposal (including their CV) to: `pypy-z at python.org`_ The deadline for this initial round of proposals is February 1, 2013. .. _`pypy-z at python.org`: mailto:pypy-z at python.org Cheers, fijal and the PyPy team From robert.zaremba at zoho.com Sat Jan 19 17:16:55 2013 From: robert.zaremba at zoho.com (Robert Zaremba) Date: Sat, 19 Jan 2013 17:16:55 +0100 Subject: [pypy-dev] python bytecode optimization. Message-ID: <19985668.84lockz7SM@rz> Hi, I really like the work you do and try to support by trying to make all my python project runing under PyPy, being PyPy proponent, convince others to PyPy, putting some founding, talk about PyPy. I try to learn PyPy architecture and someday contribute (if I got more free time). I know that PyPy bind a lot of optimizations. My question is how it looks from bytecode interpretation (so without JITing)? I learned about AST optimizers for CPython (like https://bitbucket.org/haypo/astoptimizer, hotpy). Does PyPy bytcode interpreter implements some of them on bytecode level? I know that PyPy produce similar bytecode as CPython (http://doc.pypy.org/en/latest/architecture.html#the-python-interpreter), so I guess PyPy doesn't introduce such optymalizations. But maybe on some other interpreter level? PyPy fail on some astoptimizer tests. It is pure Python, but manipulates AST. I make a little introspection and i think, the reason is due to small bytecode differences? The AST (resulted and expected) looks similar, by the PyPy one is simpler. From fijall at gmail.com Sun Jan 20 14:59:35 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 20 Jan 2013 15:59:35 +0200 Subject: [pypy-dev] RPython pypy split merge Message-ID: We are in the progress of splitting the RPython Toolchain from PyPy and merged the split-rpython branch into default. Existing code needs to be changed, usually just import renames. Most importantly: pypy.annotation -> rpython.annotator pypy.objspace.flow -> rpython.flowspace pypy.jit -> rpython.jit pypy.rlib -> rpython.rlib pypy.rpython -> rpython.rtyper pypy.translator -> rpython.translator Other changes: - py.py is now called pyinteractive.py. - pypy.module.signal.interp_signal is now rpython.rlib.rsignal - autopath.py has been killed I hope I didn't break your code too much! - Alexander Hesse (Aquana) From amauryfa at gmail.com Mon Jan 21 12:24:23 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 21 Jan 2013 12:24:23 +0100 Subject: [pypy-dev] python bytecode optimization. In-Reply-To: <19985668.84lockz7SM@rz> References: <19985668.84lockz7SM@rz> Message-ID: Hi, 2013/1/19 Robert Zaremba > Hi, > I really like the work you do and try to support by trying to make all my > python project runing under PyPy, being PyPy proponent, convince others to > PyPy, putting some founding, talk about PyPy. > Thank you, and good luck! > I try to learn PyPy architecture and someday contribute (if I got more free > time). I know that PyPy bind a lot of optimizations. > My question is how it looks from bytecode interpretation (so without > JITing)? > I learned about AST optimizers for CPython (like > https://bitbucket.org/haypo/astoptimizer, hotpy). > Does PyPy bytcode interpreter implements some of them on bytecode level? I > know that PyPy produce similar bytecode as CPython > (http://doc.pypy.org/en/latest/architecture.html#the-python-interpreter), > so I > guess PyPy doesn't introduce such optymalizations. But maybe on some other > interpreter level? > IIRC, PyPy used an AST compiler before CPython did. Most (all?) code optimizations are already performed at the AST level, see https://bitbucket.org/pypy/pypy/src/default/pypy/interpreter/astcompiler/optimize.py Cpython on the other hand optimizes the bytecode itself (in Python/peephole.c) PyPy fail on some astoptimizer tests. It is pure Python, but manipulates > AST. > I make a little introspection and i think, the reason is due to small > bytecode > differences? The AST (resulted and expected) looks similar, by the PyPy > one is > simpler. > astoptimizer is similar to the one built in PyPy, but much more aggressive, and can sometimes change semantics. For example, it assume that builtin functions are not shadowed. It can be interesting in many cases, but will produce different behavior in some cases. Most of the failures in astoptimize tests are shallow: When compiling from an ast tree, PyPy will optimize it and modify it in-place (CPython doesn't) I modified a bit these tests to add a call to compile() on the ast used for comparison, and most tests pass now. They found a crash too, around the empty set(); I fixed it in my workspace, will try to submit it tonight. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Mon Jan 21 12:38:07 2013 From: arigo at tunes.org (Armin Rigo) Date: Mon, 21 Jan 2013 12:38:07 +0100 Subject: [pypy-dev] python bytecode optimization. In-Reply-To: References: <19985668.84lockz7SM@rz> Message-ID: Hi, We should also mention that all, or almost all, of these optimizations have limited benefits in PyPy: as soon as the JIT kicks in, they are entirely pointless. They have a limited benefit anyway in the sense that they make the interpreter part marginally faster. A bient?t, Armin. From fijall at gmail.com Mon Jan 21 12:52:51 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 21 Jan 2013 13:52:51 +0200 Subject: [pypy-dev] python bytecode optimization. In-Reply-To: References: <19985668.84lockz7SM@rz> Message-ID: On Mon, Jan 21, 2013 at 1:38 PM, Armin Rigo wrote: > Hi, > > We should also mention that all, or almost all, of these optimizations > have limited benefits in PyPy: as soon as the JIT kicks in, they are > entirely pointless. They have a limited benefit anyway in the sense > that they make the interpreter part marginally faster. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev As long as they're correct (most of them arent) From robert.zaremba at zoho.com Mon Jan 21 12:59:30 2013 From: robert.zaremba at zoho.com (Robert Zaremba) Date: Mon, 21 Jan 2013 12:59:30 +0100 Subject: [pypy-dev] python bytecode optimization. In-Reply-To: References: <19985668.84lockz7SM@rz> Message-ID: <2123959.3ZPPmGnofu@rz> Thanks for all responses! I will try to compare both imlementations. I'm aware about JIT (its own optimizations in conjunction with gcc ones), so my question was only about interpreter stage (non JITed path). Cheers. From exarkun at twistedmatrix.com Thu Jan 24 20:32:02 2013 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 24 Jan 2013 19:32:02 -0000 Subject: [pypy-dev] FreeBSD 9-STABLE Message-ID: <20130124193202.3644.611174264.divmod.xquotient.228@localhost6.localdomain6> Hello all, Looking at PyPy's buildbot, it appears that the current FreeBSD builder is a FreeBSD 7 system. I'm curious who out there is interested in FreeBSD 7 in particular, and whether anyone is interested in newer versions (say, FreeBSD 9, recently released). If there's interest, I think I can offer a FreeBSD 9 builder, either to replace or augment the current FreeBSD builder. Jean-Paul From kennylevinsen at gmail.com Thu Jan 24 20:38:16 2013 From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen) Date: Thu, 24 Jan 2013 20:38:16 +0100 Subject: [pypy-dev] FreeBSD 9-STABLE In-Reply-To: <20130124193202.3644.611174264.divmod.xquotient.228@localhost6.localdomain6> References: <20130124193202.3644.611174264.divmod.xquotient.228@localhost6.localdomain6> Message-ID: Hi, I'm personally interested in BSD support, as I will soon start playing with BSD again (There's some interesting things brewing - like ArchBSD), but it might not be enough to justify a dedicated buildslave? Just a heads-up that there is at least slight interest in BSD support! Kenny Levinsen // joushou On Jan 24, 2013, at 8:32 PM, exarkun at twistedmatrix.com wrote: > Hello all, > > Looking at PyPy's buildbot, it appears that the current FreeBSD builder is a FreeBSD 7 system. I'm curious who out there is interested in FreeBSD 7 in particular, and whether anyone is interested in newer versions (say, FreeBSD 9, recently released). > > If there's interest, I think I can offer a FreeBSD 9 builder, either to replace or augment the current FreeBSD builder. > > Jean-Paul > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From skip at pobox.com Sun Jan 27 15:22:00 2013 From: skip at pobox.com (Skip Montanaro) Date: Sun, 27 Jan 2013 08:22:00 -0600 Subject: [pypy-dev] Extending include path when translating? Message-ID: I'm back messing around, trying to get PyPy going on Solaris. I worked around one issue to get it to use the -std=c99 flag with GCC. I stumbled on a bug in ctypes/util.py and found a patch which solves that problem: http://bugs.python.org/issue5289 My latest issue is slightly deeper in translation: [translation:WARNING] The module '_hashlib' is disabled [translation:WARNING] because importing pypy.module._ssl.interp_ssl raised CompilationError [translation:WARNING] CompilationError(out=""" [translation:WARNING] CompilationError(err=""" [translation:WARNING] platcheck_6.c:29:25: error: openssl/ssl.h: No such file or directory [translation:WARNING] platcheck_6.c:30:25: error: openssl/err.h: No such file or directory [translation:WARNING] platcheck_6.c:31:26: error: openssl/rand.h: No such file or directory [translation:WARNING] platcheck_6.c:32:25: error: openssl/evp.h: No such file or directory [translation:WARNING] platcheck_6.c:33:30: error: openssl/ossl_typ.h: No such file or directory [translation:WARNING] platcheck_6.c:34:28: error: openssl/x509v3.h: No such file or directory [translation:WARNING] """)""") [translation:WARNING] The module 'pyexpat' is disabled [translation:WARNING] because importing pypy.module.pyexpat.interp_pyexpat raised CompilationError [translation:WARNING] CompilationError(out=""" [translation:WARNING] CompilationError(err=""" [translation:WARNING] platcheck_7.c:29:19: error: expat.h: No such file or directory [translation:WARNING] """)""") [translation:WARNING] The module '_ssl' is disabled [translation:WARNING] because importing pypy.module._ssl.interp_ssl raised CompilationError [translation:WARNING] CompilationError(out=""" [translation:WARNING] CompilationError(err=""" [translation:WARNING] platcheck_14.c:29:25: error: openssl/ssl.h: No such file or directory [translation:WARNING] platcheck_14.c:30:25: error: openssl/err.h: No such file or directory [translation:WARNING] platcheck_14.c:31:26: error: openssl/rand.h: No such file or directory [translation:WARNING] platcheck_14.c:32:25: error: openssl/evp.h: No such file or directory [translation:WARNING] platcheck_14.c:33:30: error: openssl/ossl_typ.h: No such file or directory [translation:WARNING] platcheck_14.c:34:28: error: openssl/x509v3.h: No such file or directory [translation:WARNING] """)""") I actually have OpenSSL and Expat available, but their headers aren't in standard locations like /usr/include. Is there a way to extend the include search path (and probably library search path) so the translator can find them in their non-standard locations? Thanks, Skip Montanaro From fijall at gmail.com Sun Jan 27 15:47:16 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 27 Jan 2013 16:47:16 +0200 Subject: [pypy-dev] Extending include path when translating? In-Reply-To: References: Message-ID: On Sun, Jan 27, 2013 at 4:22 PM, Skip Montanaro wrote: > I'm back messing around, trying to get PyPy going on Solaris. I > worked around one issue to get it to use the -std=c99 flag with GCC. > I stumbled on a bug in ctypes/util.py and found a patch which solves > that problem: > > http://bugs.python.org/issue5289 > > My latest issue is slightly deeper in translation: > > [translation:WARNING] The module '_hashlib' is disabled > [translation:WARNING] because importing pypy.module._ssl.interp_ssl > raised CompilationError > [translation:WARNING] CompilationError(out=""" > [translation:WARNING] CompilationError(err=""" > [translation:WARNING] platcheck_6.c:29:25: error: > openssl/ssl.h: No such file or directory > [translation:WARNING] platcheck_6.c:30:25: error: > openssl/err.h: No such file or directory > [translation:WARNING] platcheck_6.c:31:26: error: > openssl/rand.h: No such file or directory > [translation:WARNING] platcheck_6.c:32:25: error: > openssl/evp.h: No such file or directory > [translation:WARNING] platcheck_6.c:33:30: error: > openssl/ossl_typ.h: No such file or directory > [translation:WARNING] platcheck_6.c:34:28: error: > openssl/x509v3.h: No such file or directory > [translation:WARNING] """)""") > [translation:WARNING] The module 'pyexpat' is disabled > [translation:WARNING] because importing > pypy.module.pyexpat.interp_pyexpat raised CompilationError > [translation:WARNING] CompilationError(out=""" > [translation:WARNING] CompilationError(err=""" > [translation:WARNING] platcheck_7.c:29:19: error: > expat.h: No such file or directory > [translation:WARNING] """)""") > [translation:WARNING] The module '_ssl' is disabled > [translation:WARNING] because importing pypy.module._ssl.interp_ssl > raised CompilationError > [translation:WARNING] CompilationError(out=""" > [translation:WARNING] CompilationError(err=""" > [translation:WARNING] platcheck_14.c:29:25: error: > openssl/ssl.h: No such file or directory > [translation:WARNING] platcheck_14.c:30:25: error: > openssl/err.h: No such file or directory > [translation:WARNING] platcheck_14.c:31:26: error: > openssl/rand.h: No such file or directory > [translation:WARNING] platcheck_14.c:32:25: error: > openssl/evp.h: No such file or directory > [translation:WARNING] platcheck_14.c:33:30: error: > openssl/ossl_typ.h: No such file or directory > [translation:WARNING] platcheck_14.c:34:28: error: > openssl/x509v3.h: No such file or directory > [translation:WARNING] """)""") > > I actually have OpenSSL and Expat available, but their headers aren't > in standard locations like /usr/include. Is there a way to extend the > include search path (and probably library search path) so the > translator can find them in their non-standard locations? Yes. This should be sorted by translator/platform/solaris.py I would say (that should include the correct paths). Generally it's in ExternalCompilationInfo of libraries, so it would be in pypy/module/pyexpat/interp_pyexpat.py at the top (which just uses platform-dependent library paths) > > Thanks, > > Skip Montanaro > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From kirk.liberty at gmail.com Mon Jan 28 23:37:31 2013 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Mon, 28 Jan 2013 17:37:31 -0500 Subject: [pypy-dev] Translating safely (without executing) Message-ID: Hello all, Is it possible to use PyPy's translation tool without having it execute the program? I think this would be important for programs with infinite loops, or ones which could do potentially dangerous things ie. when trying to translate a program to control a GPU fan, the fan was commanded off during a high work load. Thank you, Kirk From Ronny.Pfannschmidt at gmx.de Tue Jan 29 00:35:26 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 29 Jan 2013 00:35:26 +0100 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: Message-ID: <51070B3E.8010808@gmx.de> Hi Kirk, could you please provide more context? the translator does not execute the actual rpython program as code, just the surrounding imports (using python to metaprogramm the rpython) so its likely that the cpu fan off thing is a missunderstanding best, Ronny On 01/28/2013 11:37 PM, Kirk Liberty wrote: > Hello all, > > Is it possible to use PyPy's translation tool without having it > execute the program? I think this would be important for programs with > infinite loops, or ones which could do potentially dangerous things > ie. when trying to translate a program to control a GPU fan, the fan > was commanded off during a high work load. > > Thank you, > Kirk > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev From kirk.liberty at gmail.com Tue Jan 29 00:50:19 2013 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Mon, 28 Jan 2013 18:50:19 -0500 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: <51070B3E.8010808@gmx.de> References: <51070B3E.8010808@gmx.de> Message-ID: ronny, Well the fan control command turned out not to be in rpython, so it failed to translate, but it still stopped the fan first. Here's an example of a program that would not translate because it just enters an infinite loop. from time import sleep def main(argv): while True: print "looping" sleep(1) return 0 def target(driver,args): return main,None main(1) command to translate > python ~/pypy/translator/goal/translate.py --batch --output loop loop.py Regards, Kirk On Mon, Jan 28, 2013 at 6:35 PM, Ronny Pfannschmidt wrote: > Hi Kirk, > > could you please provide more context? > > the translator does not execute the actual rpython program as code, just the > surrounding imports (using python to metaprogramm the rpython) > > so its likely that the cpu fan off thing is a missunderstanding > > best, Ronny > > > On 01/28/2013 11:37 PM, Kirk Liberty wrote: >> >> Hello all, >> >> Is it possible to use PyPy's translation tool without having it >> execute the program? I think this would be important for programs with >> infinite loops, or ones which could do potentially dangerous things >> ie. when trying to translate a program to control a GPU fan, the fan >> was commanded off during a high work load. >> >> Thank you, >> Kirk >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > From Ronny.Pfannschmidt at gmx.de Tue Jan 29 00:55:31 2013 From: Ronny.Pfannschmidt at gmx.de (Ronny Pfannschmidt) Date: Tue, 29 Jan 2013 00:55:31 +0100 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: <51070B3E.8010808@gmx.de> Message-ID: <51070FF3.4000007@gmx.de> Hi Kirk, you need to check for __name__ == '__main__' before the last line rpython does load code by importing and the last line will always run the program, guard it with a condition, and it will work fine best, Ronny On 01/29/2013 12:50 AM, Kirk Liberty wrote: > ronny, > > Well the fan control command turned out not to be in rpython, so it > failed to translate, but it still stopped the fan first. Here's an > example of a program that would not translate because it just enters > an infinite loop. > > from time import sleep > def main(argv): > while True: > print "looping" > sleep(1) > return 0 > def target(driver,args): > return main,None > main(1) > > command to translate> python ~/pypy/translator/goal/translate.py > --batch --output loop loop.py > > Regards, > Kirk > > On Mon, Jan 28, 2013 at 6:35 PM, Ronny Pfannschmidt > wrote: >> Hi Kirk, >> >> could you please provide more context? >> >> the translator does not execute the actual rpython program as code, just the >> surrounding imports (using python to metaprogramm the rpython) >> >> so its likely that the cpu fan off thing is a missunderstanding >> >> best, Ronny >> >> >> On 01/28/2013 11:37 PM, Kirk Liberty wrote: >>> >>> Hello all, >>> >>> Is it possible to use PyPy's translation tool without having it >>> execute the program? I think this would be important for programs with >>> infinite loops, or ones which could do potentially dangerous things >>> ie. when trying to translate a program to control a GPU fan, the fan >>> was commanded off during a high work load. >>> >>> Thank you, >>> Kirk >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >> >> From fijall at gmail.com Tue Jan 29 08:49:26 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 29 Jan 2013 09:49:26 +0200 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: <51070FF3.4000007@gmx.de> References: <51070B3E.8010808@gmx.de> <51070FF3.4000007@gmx.de> Message-ID: On Tue, Jan 29, 2013 at 1:55 AM, Ronny Pfannschmidt wrote: > Hi Kirk, > > you need to check for > __name__ == '__main__' before the last line > > rpython does load code by importing > and the last line will always run the program, > guard it with a condition, and it will work fine > > best, Ronny > > > > On 01/29/2013 12:50 AM, Kirk Liberty wrote: >> >> ronny, >> >> Well the fan control command turned out not to be in rpython, so it >> failed to translate, but it still stopped the fan first. Here's an >> example of a program that would not translate because it just enters >> an infinite loop. >> >> from time import sleep >> def main(argv): >> while True: >> print "looping" >> sleep(1) >> return 0 >> def target(driver,args): >> return main,None >> main(1) >> >> command to translate> python ~/pypy/translator/goal/translate.py >> --batch --output loop loop.py >> >> Regards, >> Kirk >> >> On Mon, Jan 28, 2013 at 6:35 PM, Ronny Pfannschmidt >> wrote: >>> >>> Hi Kirk, >>> >>> could you please provide more context? >>> >>> the translator does not execute the actual rpython program as code, just >>> the >>> surrounding imports (using python to metaprogramm the rpython) >>> >>> so its likely that the cpu fan off thing is a missunderstanding >>> >>> best, Ronny >>> >>> >>> On 01/28/2013 11:37 PM, Kirk Liberty wrote: >>>> >>>> >>>> Hello all, >>>> >>>> Is it possible to use PyPy's translation tool without having it >>>> execute the program? I think this would be important for programs with >>>> infinite loops, or ones which could do potentially dangerous things >>>> ie. when trying to translate a program to control a GPU fan, the fan >>>> was commanded off during a high work load. >>>> >>>> Thank you, >>>> Kirk >>>> _______________________________________________ >>>> pypy-dev mailing list >>>> pypy-dev at python.org >>>> http://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >>> > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > http://mail.python.org/mailman/listinfo/pypy-dev Hi Kirk. Python is a meta-programming language for RPython, so no absolutely impossible. That said, why would you want to translate such programs? From kirk.liberty at gmail.com Tue Jan 29 10:03:43 2013 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Tue, 29 Jan 2013 04:03:43 -0500 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: <51070FF3.4000007@gmx.de> References: <51070B3E.8010808@gmx.de> <51070FF3.4000007@gmx.de> Message-ID: Ronny, I hadn't really thought of it as a program being imported, rather than just read as a text file. I'll add the check in my programs, thank you. Regards, Kirk On Mon, Jan 28, 2013 at 6:55 PM, Ronny Pfannschmidt wrote: > Hi Kirk, > > you need to check for > __name__ == '__main__' before the last line > > rpython does load code by importing > and the last line will always run the program, > guard it with a condition, and it will work fine > > best, Ronny > > > > On 01/29/2013 12:50 AM, Kirk Liberty wrote: >> >> ronny, >> >> Well the fan control command turned out not to be in rpython, so it >> failed to translate, but it still stopped the fan first. Here's an >> example of a program that would not translate because it just enters >> an infinite loop. >> >> from time import sleep >> def main(argv): >> while True: >> print "looping" >> sleep(1) >> return 0 >> def target(driver,args): >> return main,None >> main(1) >> >> command to translate> python ~/pypy/translator/goal/translate.py >> --batch --output loop loop.py >> >> Regards, >> Kirk >> >> On Mon, Jan 28, 2013 at 6:35 PM, Ronny Pfannschmidt >> wrote: >>> >>> Hi Kirk, >>> >>> could you please provide more context? >>> >>> the translator does not execute the actual rpython program as code, just >>> the >>> surrounding imports (using python to metaprogramm the rpython) >>> >>> so its likely that the cpu fan off thing is a missunderstanding >>> >>> best, Ronny >>> >>> >>> On 01/28/2013 11:37 PM, Kirk Liberty wrote: >>>> >>>> >>>> Hello all, >>>> >>>> Is it possible to use PyPy's translation tool without having it >>>> execute the program? I think this would be important for programs with >>>> infinite loops, or ones which could do potentially dangerous things >>>> ie. when trying to translate a program to control a GPU fan, the fan >>>> was commanded off during a high work load. >>>> >>>> Thank you, >>>> Kirk >>>> _______________________________________________ >>>> pypy-dev mailing list >>>> pypy-dev at python.org >>>> http://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >>> > From kirk.liberty at gmail.com Tue Jan 29 10:07:43 2013 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Tue, 29 Jan 2013 04:07:43 -0500 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: <51070B3E.8010808@gmx.de> <51070FF3.4000007@gmx.de> Message-ID: Maciej, I could say I'm being stingy about CPU cycles since the program will be running continuously, but mostly "just because I can", or in this case can't. I think translation is pretty cool, and I've been trying to see what different things I can apply it to. What do you mean by "Python is a meta-programming language for RPython"? Regards, Kirk On Tue, Jan 29, 2013 at 2:49 AM, Maciej Fijalkowski wrote: > On Tue, Jan 29, 2013 at 1:55 AM, Ronny Pfannschmidt > wrote: >> Hi Kirk, >> >> you need to check for >> __name__ == '__main__' before the last line >> >> rpython does load code by importing >> and the last line will always run the program, >> guard it with a condition, and it will work fine >> >> best, Ronny >> >> >> >> On 01/29/2013 12:50 AM, Kirk Liberty wrote: >>> >>> ronny, >>> >>> Well the fan control command turned out not to be in rpython, so it >>> failed to translate, but it still stopped the fan first. Here's an >>> example of a program that would not translate because it just enters >>> an infinite loop. >>> >>> from time import sleep >>> def main(argv): >>> while True: >>> print "looping" >>> sleep(1) >>> return 0 >>> def target(driver,args): >>> return main,None >>> main(1) >>> >>> command to translate> python ~/pypy/translator/goal/translate.py >>> --batch --output loop loop.py >>> >>> Regards, >>> Kirk >>> >>> On Mon, Jan 28, 2013 at 6:35 PM, Ronny Pfannschmidt >>> wrote: >>>> >>>> Hi Kirk, >>>> >>>> could you please provide more context? >>>> >>>> the translator does not execute the actual rpython program as code, just >>>> the >>>> surrounding imports (using python to metaprogramm the rpython) >>>> >>>> so its likely that the cpu fan off thing is a missunderstanding >>>> >>>> best, Ronny >>>> >>>> >>>> On 01/28/2013 11:37 PM, Kirk Liberty wrote: >>>>> >>>>> >>>>> Hello all, >>>>> >>>>> Is it possible to use PyPy's translation tool without having it >>>>> execute the program? I think this would be important for programs with >>>>> infinite loops, or ones which could do potentially dangerous things >>>>> ie. when trying to translate a program to control a GPU fan, the fan >>>>> was commanded off during a high work load. >>>>> >>>>> Thank you, >>>>> Kirk >>>>> _______________________________________________ >>>>> pypy-dev mailing list >>>>> pypy-dev at python.org >>>>> http://mail.python.org/mailman/listinfo/pypy-dev >>>> >>>> >>>> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > Hi Kirk. > > Python is a meta-programming language for RPython, so no absolutely impossible. > > That said, why would you want to translate such programs? From amauryfa at gmail.com Tue Jan 29 10:16:11 2013 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 29 Jan 2013 10:16:11 +0100 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: <51070B3E.8010808@gmx.de> <51070FF3.4000007@gmx.de> Message-ID: 2013/1/29 Kirk Liberty > Maciej, > > I could say I'm being stingy about CPU cycles since the program will > be running continuously, but mostly "just because I can", or in this > case can't. I think translation is pretty cool, and I've been trying > to see what different things I can apply it to. > > What do you mean by "Python is a meta-programming language for RPython"? > Please read: https://pypy.readthedocs.org/en/latest/you-want-to-help.html?highlight=meta#architecture -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Jan 29 10:13:20 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 29 Jan 2013 11:13:20 +0200 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: <51070B3E.8010808@gmx.de> <51070FF3.4000007@gmx.de> Message-ID: On Tue, Jan 29, 2013 at 11:07 AM, Kirk Liberty wrote: > Maciej, > > I could say I'm being stingy about CPU cycles since the program will > be running continuously, but mostly "just because I can", or in this > case can't. I think translation is pretty cool, and I've been trying > to see what different things I can apply it to. I would seriously suggest pypy JIT here. It can sometimes even produce better results. > > What do you mean by "Python is a meta-programming language for RPython"? you can run some python before compiling RPython. we use all the tricks to say exec a bunch of methods. > > Regards, > Kirk > > On Tue, Jan 29, 2013 at 2:49 AM, Maciej Fijalkowski wrote: >> On Tue, Jan 29, 2013 at 1:55 AM, Ronny Pfannschmidt >> wrote: >>> Hi Kirk, >>> >>> you need to check for >>> __name__ == '__main__' before the last line >>> >>> rpython does load code by importing >>> and the last line will always run the program, >>> guard it with a condition, and it will work fine >>> >>> best, Ronny >>> >>> >>> >>> On 01/29/2013 12:50 AM, Kirk Liberty wrote: >>>> >>>> ronny, >>>> >>>> Well the fan control command turned out not to be in rpython, so it >>>> failed to translate, but it still stopped the fan first. Here's an >>>> example of a program that would not translate because it just enters >>>> an infinite loop. >>>> >>>> from time import sleep >>>> def main(argv): >>>> while True: >>>> print "looping" >>>> sleep(1) >>>> return 0 >>>> def target(driver,args): >>>> return main,None >>>> main(1) >>>> >>>> command to translate> python ~/pypy/translator/goal/translate.py >>>> --batch --output loop loop.py >>>> >>>> Regards, >>>> Kirk >>>> >>>> On Mon, Jan 28, 2013 at 6:35 PM, Ronny Pfannschmidt >>>> wrote: >>>>> >>>>> Hi Kirk, >>>>> >>>>> could you please provide more context? >>>>> >>>>> the translator does not execute the actual rpython program as code, just >>>>> the >>>>> surrounding imports (using python to metaprogramm the rpython) >>>>> >>>>> so its likely that the cpu fan off thing is a missunderstanding >>>>> >>>>> best, Ronny >>>>> >>>>> >>>>> On 01/28/2013 11:37 PM, Kirk Liberty wrote: >>>>>> >>>>>> >>>>>> Hello all, >>>>>> >>>>>> Is it possible to use PyPy's translation tool without having it >>>>>> execute the program? I think this would be important for programs with >>>>>> infinite loops, or ones which could do potentially dangerous things >>>>>> ie. when trying to translate a program to control a GPU fan, the fan >>>>>> was commanded off during a high work load. >>>>>> >>>>>> Thank you, >>>>>> Kirk >>>>>> _______________________________________________ >>>>>> pypy-dev mailing list >>>>>> pypy-dev at python.org >>>>>> http://mail.python.org/mailman/listinfo/pypy-dev >>>>> >>>>> >>>>> >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> http://mail.python.org/mailman/listinfo/pypy-dev >> >> Hi Kirk. >> >> Python is a meta-programming language for RPython, so no absolutely impossible. >> >> That said, why would you want to translate such programs? From arigo at tunes.org Wed Jan 30 09:25:42 2013 From: arigo at tunes.org (Armin Rigo) Date: Wed, 30 Jan 2013 09:25:42 +0100 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: Message-ID: Hi Kirk, On Mon, Jan 28, 2013 at 11:37 PM, Kirk Liberty wrote: > Is it possible to use PyPy's translation tool without having it > execute the program? As you got it, the answer to that is no. I updated https://pypy.readthedocs.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython . A bient?t, Armin. From kirk.liberty at gmail.com Wed Jan 30 18:51:15 2013 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Wed, 30 Jan 2013 12:51:15 -0500 Subject: [pypy-dev] Translating safely (without executing) In-Reply-To: References: Message-ID: Armin, I read your addition to the FAQ, I think I understand now the purpose of RPython and why I should just use the PyPy JIT for my programs. Kirk On Jan 30, 2013 3:26 AM, "Armin Rigo" wrote: > Hi Kirk, > > On Mon, Jan 28, 2013 at 11:37 PM, Kirk Liberty > wrote: > > Is it possible to use PyPy's translation tool without having it > > execute the program? > > As you got it, the answer to that is no. I updated > > https://pypy.readthedocs.org/en/latest/faq.html#do-i-have-to-rewrite-my-programs-in-rpython > . > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu Jan 31 19:11:56 2013 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 31 Jan 2013 20:11:56 +0200 Subject: [pypy-dev] (unexpectedly) bad performance of stackless and greenlet on PyPy In-Reply-To: References: Message-ID: On Sun, Dec 30, 2012 at 5:53 PM, Maciej Fijalkowski wrote: > On Sun, Dec 30, 2012 at 2:24 PM, Denis Bilenko wrote: >> Hi! >> >> I've micro-benchmarked stackless and greenlet module in PyPy and was >> quite surprised with the results. According to this benchmark >> stackless.channel is 100x times slower on PyPy (vs Stackless) and >> greenlet is 20x times slower on PyPy (vs CPython). >> >> https://gist.github.com/4412582 >> >> Is this a bug in benchmark (or PyPy) or is it the expected current >> state of stackless features on PyPy? >> >> Cheers, >> Denis. >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> http://mail.python.org/mailman/listinfo/pypy-dev > > This is at least partly expected because stackless and/or greenlets > disable the JIT. It probably shouldn't be 100x though. There is an > ongoing work to make this reason go away and then we can look what's > left. > > Cheers, > fijal Hi Denis. I'm pleased to say that we're on the right track to merge JIT + stackless branch to pypy From john.m.camara at gmail.com Thu Jan 31 23:01:00 2013 From: john.m.camara at gmail.com (John Camara) Date: Thu, 31 Jan 2013 17:01:00 -0500 Subject: [pypy-dev] Parallella open hardware platform Message-ID: A couple of days ago I heard about the Parallella [1] project which is an open hardware platform similar to the Raspberry Pi but with much higher capabilities. It has a Zynq Z-7010 which has both a dual core ARM A9 (800 MHz) processor and a Artix-7 FPGA, a 16 core Epiphany multicore accelerator, 1GB ram (see [2] for more info) and currently boots up in Ubuntu. The goal of the Parallella project is to develop an open parallel hardware platform and development tools. Recently they announced support for Python with Mark Dewing [3] leading the effort. I had asked Mark if he considered PyPy but at this time he doesn't have time for this investigation and he reposted my comment on the forum [4] with a couple of question. Maybe one of you could answer them. Working with the Parallella project maybe a good opportunity for the PyPy project from both a PR perspective and as well as the technical challenges it would present. On the technical side it would give the opportunity to test STM on a reasonable number of cores while also dealing with cores from different architectures (ARM and Epiphany). I could see all the JITting occurring on the ARM cores with it producing output for both architectures based on which type of core STM decides to use for a chunk of work to execute on. Of course there is also the challenge of bridging between the 2 architectures. Maybe even some of the more expensive STM operations could be offloaded to the FPGA or even a limited amount of very hot sections of code could be JITted to the FPGA (although this might be more work than its worth). >From a PR perspective PyPy needs to excel at some niche market so that the PyPy platform can take off. When PyPy started concentrating on the scientific market with increasing support for Numpy I thought this would be the niche market that would get PyPy to take off. But there have been a couple of issue with this approach. There is a tremendous amount of work that needs to be done so that PyPy can look attractive to this niche market. It requires supporting both NumPy and SciPy and their was an expectation that if PyPy supports NumPy others would come to help out with the SciPy support. The problem is that there doesn't seam to be many who are eager to pitch in for the SciPy effort and there also has not been a whole lot willing to help will the ongoing NumPy work. I think in general the ratio of people who use NumPy and SciPy to those willing to contribute is quite small. So the idea of going after this market was a good idea and can definitely have the opportunity to showing the strength of PyPy project it hasn't done much to improve the image of the PyPy project. It also doesn't help that there is some commercial interests that have popped up recently that have decided to play hard ball against PyPy by spreading FUD. Unlike the Raspberry Pi hardware which can only support hobbyist the Parallella hardware can support both hobbyists and commercial interests. They cost $100 which is more than the $35 for Raspberry Pi but still within reach of most hobbyists and they didn't cut out the many features that are needed for commercial interests. The Parallella project raised nearly $0.9 million on kickstarter [5] for the project with nearly 5000 backers. Since many who will use the Parallella hardware also have experience on embedded systems they and are more likely used to writing low level code in assembly, FPGAs, and even lots of C code and I'm sure have hit many issues with programming in parallel/multithreaded and would welcome a better developer experience. I bet many of them would be willing to contribute both financially and time to supporting such an effort. I believe the Architecture of PyPy could lend it self to becoming the core of such a development system and would allow Python to be used in this space. This could provide a lot of good PR for the PyPy project. Now I'm not saying PyPy shouldn't devote any more time to supporting NumPy as I'm sure when PyPy has very good support for both NumPy and SciPy it's going to be a very good day for all Python supporters. I just think that the PyPy team needs to think about a strategy that in the end will help its PR and gain support from a much larger community. This project is doing a lot of good things technically and now it just needs to get the attention of the development community at large. Now I can't predict if working with the Parallella project would be the break though in PR that PyPy needs but it's at least an option that's out there. BTW I don't have any commercial interests in the Parallella project. If some time in the future I use their hardware it would likely be as a hobbyist and it would be nice to program it in Python. My real objective of this post to see the PyPy project gain wider interest as it would be a good thing for Python. [1] - http://www.parallella.org/ [2] - http://www.parallella.org/board/ [3] - http://forums.parallella.org/memberlist.php?mode=viewprofile&u=3344 [4] - http://forums.parallella.org/viewtopic.php?f=26&t=139 [5] - http://www.kickstarter.com/projects/adapteva/parallella-a-supercomputer-for-everyone John -------------- next part -------------- An HTML attachment was scrubbed... URL: