From lac at openend.se Mon May 4 01:26:53 2015 From: lac at openend.se (lac at openend.se) Date: Mon, 04 May 2015 01:26:53 +0200 Subject: [pypy-dev] (no subject) Message-ID: <201505032326.t43NQrRm019774@fido.openend.se> Hello gang. Mario Reingart has been trying to internationalise CPython since at least 2012. Here is his lastest proposal, for GSOC. http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/reingart/5634387206995968 The PSF declined to fund it. He's written a PEP about it, which apparantly isn't in the list of PEPs anywhere, and has had this discussion, such as it was on the python-dev mailing list. http://bugs.python.org/issue16344 seems to have the arguments against having it put into Cpython. The arguments against seem very weak to me. My idea is that it would be good to intenationalise PyPy and then we can capture all the people who want to teach Python in the non-english speaking world. Mariano already has written a book about using web2py, but as far as I know we are perfectly compliant with this. Mariano didn't know anything about PyPy when I wrote to him about it. The idea that it was a completely Python compliant version was news to him... an indication that we lack international communication presence. This could be a way out of obscurity. Poor old Mariano has been fighting this fight since before 2012. He is sad and unhappy. Mostly he is sad and unhappy because people won't discuss what he wants, or move from -- ideals fine, implementation well, would be better this way --- or any sort of reasonable discussion. 'Hung out to dry' comes to mind. I sincerely suspect that the people here will support this idea, but may argue that in pypy there are better ways to do it, and, well we will get it done. But in any case, please be kind to Mariano, since he has already suffered enough for the cause. ps -- funding it comes later. I'd like to get Mariano paid. But I'd like to get some sort of idea that we want to go forward with this before I make a different note on how I would like to approach funding this. Take care all, Laura From steve at pearwood.info Mon May 4 05:27:44 2015 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 4 May 2015 13:27:44 +1000 Subject: [pypy-dev] (no subject) In-Reply-To: <201505032326.t43NQrRm019774@fido.openend.se> References: <201505032326.t43NQrRm019774@fido.openend.se> Message-ID: <20150504032743.GV5663@ando.pearwood.info> On Mon, May 04, 2015 at 01:26:53AM +0200, lac at openend.se wrote: > Hello gang. > > Mario Reingart has been trying to internationalise CPython > since at least 2012. [...] I've just added this comment to the issue on the bug tracker: ==== For what it's worth, there are at least two localised versions of Python: Teuton and ChinesePython. As far as I know, ChinesePython is still in active development. Both translate the keywords and builtins, to German and Chinese respectively. I don't have a link, but I recall that Guido gave his blessing for ChinesePython to call itself a Python implementation. There's a localised, Portuguese version of Stack Overflow: http://blog.stackoverflow.com/2014/02/cant-we-all-be-reasonable-and-speak-english/ so I think that the days when all programmers must learn English are slowly fading away, just like the days when all mathematicians had to learn German. But, I agree with those who say that a PEP is necessary. There are a lot of factors to consider. (Although of course as a third-party library, no PEP would be needed.) ==== http://bugs.python.org/issue16344 > He's written a PEP about it, which apparantly isn't in the list of > PEPs anywhere, and has had this discussion, such as it was on the > python-dev mailing list. Anyone can write a PEP and have it sit on their computer's hard drive. Unless he follows the process for submitting it for discussion, it might as well not exist. > My idea is that it would be good to intenationalise > PyPy and then we can capture all the > people who want to teach Python in the non-english speaking world. [...] > Poor old Mariano has been fighting this fight since before 2012. > He is sad and unhappy. Mostly he is sad and unhappy because people > won't discuss what he wants, or move from -- ideals fine, implementation > well, would be better this way --- or any sort of reasonable discussion. > 'Hung out to dry' comes to mind. Looking at the volume of comments on the issue at the tracker link above, I don't think Mariano has been *ignored*. Possibly the conservativism of Python-Dev has struck again. There was a lot of resistance to the idea of allowing non-ASCII identifiers when that was first proposed too. My own feeling is that CPython should not internationalise error messages *yet*. But I would like to see a third-party implementation, possibly something that can be switched on and off at the interactive interpreter, or in iPython, and see how useful it is. But that's something which can be discussed in the PEP. > I sincerely suspect that the people here will support this idea, > but may argue that in pypy there are better ways to do it, and, well > we will get it done. But in any case, please be kind to Mariano, > since he has already suffered enough for the cause. > > ps -- funding it comes later. I'd like to get Mariano paid. > But I'd like to get some sort of idea that we want to go forward > with this before I make a different note on how I would like to > approach funding this. I think that if Mariano had a proof of concept, and evidence that it would be useful in practice as well as theory, then I think the PSF should be asked to consider funding this as a diversity measure. -- Steve From mikeckennedy at gmail.com Fri May 1 19:07:49 2015 From: mikeckennedy at gmail.com (Michael Kennedy) Date: Fri, 01 May 2015 17:07:49 +0000 Subject: [pypy-dev] Be on my podcast In-Reply-To: References: Message-ID: Great, sent you an invite. Let me know if it works! On Thu, Apr 30, 2015 at 9:28 AM Maciej Fijalkowski wrote: > After the 25th of May > > On Thu, Apr 30, 2015 at 6:12 PM, Michael Kennedy > wrote: > > Hi Maciej, > > > > Accents are fine. :) Do you have some time later in May (last few weeks)? > > > > Thanks, > > Michael > > > > > > On Wed, Apr 22, 2015 at 9:33 AM Maciej Fijalkowski > wrote: > >> > >> Hi Michael > >> > >> I'm sorry I did not reply, somehow missed it. Sure, I'm happy to be on > >> your podcast, beware of my thick eastern european accent though :-) > >> > >> Cheers, > >> fijal > >> > >> On Wed, Apr 15, 2015 at 8:07 PM, Michael Kennedy < > mikeckennedy at gmail.com> > >> wrote: > >> > I'd love to have you guys on my podcast, Talk Python To Me. You can > >> > learn > >> > more here: > >> > > >> > http://www.talkpythontome.com/ > >> > > >> > Interested in being a guest? Or a couple of you even? > >> > > >> > Thanks! > >> > Michael > >> > > >> > _______________________________________________ > >> > pypy-dev mailing list > >> > pypy-dev at python.org > >> > https://mail.python.org/mailman/listinfo/pypy-dev > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Mon May 4 10:00:49 2015 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 May 2015 10:00:49 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: <20150504032743.GV5663@ando.pearwood.info> References: <201505032326.t43NQrRm019774@fido.openend.se> <20150504032743.GV5663@ando.pearwood.info> Message-ID: Hi Laura, hi all, On 4 May 2015 at 05:27, Steven D'Aprano wrote: >> Mario Reingart has been trying to internationalise CPython >> since at least 2012. Fwiw, I found a 2012 text that says "since 2010". I'm absolutely +1 on the idea. I recall my own experience as a child: learning new keywords is fine --- you need to learn them anyway. (Honestly, keywords like "for" have a very specific meaning that is mostly unrelated to the general meaning of the word "for" in English.) Even "syntax error!" is fine if your programming language is BASIC and there are only a few common error messages possible. But if the book I had was in English I would not even have started. In the case of Python, there are translated books, of course, but the first difference with BASIC is that we can get commonly many more error messages. I think that allowing them to be translated is a good first step. Well, this is only my personal opinion. Possibly more relevant to this case is what already exists in the wide. In fact, gcc, bash, mercurial, .NET and MySQL are all examples where the error messages are localized (found them by quickly scanning over the pages of discussion), but where the keywords are still in English. So "there is no prior art" is plainly wrong. In fact, say what you want, but bash *is* a programming language. > I think that if Mariano had a proof of concept, and evidence that it > would be useful in practice as well as theory, then I think the PSF > should be asked to consider funding this as a diversity measure. And I believe that he has got a proof of concept (there is a patch), and plenty of evidence that it is useful in practice (there is the long list of other tools and languages where it is already done). My point of view is that the python-dev community plainly ignored him, and continues to by rejecting his GSoC proposal. I would like to push forward having a PyPy which supports his goal. If anything, as Laura said, it would be a way to push forward PyPy in the relevant communities. Teaching children and non-technical people to start a Python interpreter, even if just to play around a bit --- this is imho the best way to demystify programming. A bient?t, Armin. From eun.che.4 at gmail.com Mon May 4 09:58:42 2015 From: eun.che.4 at gmail.com (Eun Che) Date: Mon, 4 May 2015 16:58:42 +0900 Subject: [pypy-dev] PyPy on Windows 7: Icon file for pypy.exe Message-ID: Greeting. I'm recently start to study Python with PyPy, and found that pypy.exe has no icon. So, here I send a simple pypy.ico file. It's made from original PyPy logo. Could you dress pypy.exe with this? I wished to contribute via Bitbucket fork, but i'm so new to that, don't know how to do. So I had to send this mail. Thanks. Sincerely. - Eun Che -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: pypy.ico Type: image/x-icon Size: 99678 bytes Desc: not available URL: From arigo at tunes.org Mon May 4 10:37:53 2015 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 May 2015 10:37:53 +0200 Subject: [pypy-dev] PyPy on Windows 7: Icon file for pypy.exe In-Reply-To: References: Message-ID: Hi Eun, On 4 May 2015 at 09:58, Eun Che wrote: > I wished to contribute via Bitbucket fork, but i'm so new to that, don't > know how to do. So I had to send this mail. Thanks! Can you give us a link that explains how to embed this icon inside pypy.exe? Maybe it's just a command-line option of the linker command, but I wouldn't be surprized if it's more involved. A bient?t, Armin. From cfbolz at gmx.de Mon May 4 12:05:31 2015 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 4 May 2015 12:05:31 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: <201505032326.t43NQrRm019774@fido.openend.se> References: <201505032326.t43NQrRm019774@fido.openend.se> Message-ID: Hi Laura, Thanks for kicking this off. Just a brief comment: I am strongly for trying to get multi-language support into PyPy, particularly error messages and docstrings. The actual translations would have to be crowd-sourced of course, but there are various tools for that. There are a few technical questions that need discussing, but those are probably solvable. (actually on the technical side I think there is a good reason why pypy is well-positioned to implement this: our error messages are already a bit different than CPython's, so code that depends on the precise error string is anyway potentially broken on pypy.) Cheers, Carl Friedrich On May 4, 2015 01:27, wrote: > Hello gang. > > Mario Reingart has been trying to internationalise CPython > since at least 2012. > > Here is his lastest proposal, for GSOC. > > http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/reingart/5634387206995968 > > The PSF declined to fund it. > > He's written a PEP about it, which apparantly isn't in the list of > PEPs anywhere, and has had this discussion, such as it was on the > python-dev mailing list. > > http://bugs.python.org/issue16344 seems to have the arguments against > having it put into Cpython. > > The arguments against seem very weak to me. > > My idea is that it would be good to intenationalise > PyPy and then we can capture all the > people who want to teach Python in the non-english speaking world. > Mariano already has written a book about using web2py, but as far > as I know we are perfectly compliant with this. Mariano didn't > know anything about PyPy when I wrote to him about it. The idea > that it was a completely Python compliant version was news to him... > an indication that we lack international communication presence. > This could be a way out of obscurity. > > Poor old Mariano has been fighting this fight since before 2012. > He is sad and unhappy. Mostly he is sad and unhappy because people > won't discuss what he wants, or move from -- ideals fine, implementation > well, would be better this way --- or any sort of reasonable discussion. > 'Hung out to dry' comes to mind. > > I sincerely suspect that the people here will support this idea, > but may argue that in pypy there are better ways to do it, and, well > we will get it done. But in any case, please be kind to Mariano, > since he has already suffered enough for the cause. > > ps -- funding it comes later. I'd like to get Mariano paid. > But I'd like to get some sort of idea that we want to go forward > with this before I make a different note on how I would like to > approach funding this. > > Take care all, > Laura > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From samo at meluria.com Mon May 4 13:06:13 2015 From: samo at meluria.com (Samuel Villamonte) Date: Mon, 4 May 2015 06:06:13 -0500 Subject: [pypy-dev] (no subject) In-Reply-To: References: <201505032326.t43NQrRm019774@fido.openend.se> Message-ID: +1 here! May I suggest using Transifex? That way you could crowdsource translation not only to Spanish but to other languages. Some documentation is available here: http://docs.transifex.com/developer/client/ http://docs.transifex.com/developer/formats/gettext http://docs.transifex.com/faq/#the-open-source-project-definition Disclamer: 1. I know nothing about RPython nor PyPy internals, but from what I gather about the patch it could be handled through gettext .PO files, right? 2. I do have an account at Transifex and volunteer at the Coursera Global Translator community, In fact, Coursera is translating an introductory Python course: https://www.coursera.org/course/interactivepython1 Regards, 2015-05-04 5:05 GMT-05:00 Carl Friedrich Bolz : > Hi Laura, > > Thanks for kicking this off. Just a brief comment: I am strongly for > trying to get multi-language support into PyPy, particularly error messages > and docstrings. The actual translations would have to be crowd-sourced of > course, but there are various tools for that. > > There are a few technical questions that need discussing, but those are > probably solvable. > > (actually on the technical side I think there is a good reason why pypy is > well-positioned to implement this: our error messages are already a bit > different than CPython's, so code that depends on the precise error string > is anyway potentially broken on pypy.) > > Cheers, > > Carl Friedrich > On May 4, 2015 01:27, wrote: > >> Hello gang. >> >> Mario Reingart has been trying to internationalise CPython >> since at least 2012. >> >> Here is his lastest proposal, for GSOC. >> >> http://www.google-melange.com/gsoc/proposal/public/google/gsoc2015/reingart/5634387206995968 >> >> The PSF declined to fund it. >> >> He's written a PEP about it, which apparantly isn't in the list of >> PEPs anywhere, and has had this discussion, such as it was on the >> python-dev mailing list. >> >> http://bugs.python.org/issue16344 seems to have the arguments against >> having it put into Cpython. >> >> The arguments against seem very weak to me. >> >> My idea is that it would be good to intenationalise >> PyPy and then we can capture all the >> people who want to teach Python in the non-english speaking world. >> Mariano already has written a book about using web2py, but as far >> as I know we are perfectly compliant with this. Mariano didn't >> know anything about PyPy when I wrote to him about it. The idea >> that it was a completely Python compliant version was news to him... >> an indication that we lack international communication presence. >> This could be a way out of obscurity. >> >> Poor old Mariano has been fighting this fight since before 2012. >> He is sad and unhappy. Mostly he is sad and unhappy because people >> won't discuss what he wants, or move from -- ideals fine, implementation >> well, would be better this way --- or any sort of reasonable discussion. >> 'Hung out to dry' comes to mind. >> >> I sincerely suspect that the people here will support this idea, >> but may argue that in pypy there are better ways to do it, and, well >> we will get it done. But in any case, please be kind to Mariano, >> since he has already suffered enough for the cause. >> >> ps -- funding it comes later. I'd like to get Mariano paid. >> But I'd like to get some sort of idea that we want to go forward >> with this before I make a different note on how I would like to >> approach funding this. >> >> Take care all, >> Laura >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon May 4 16:44:37 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 4 May 2015 09:44:37 -0500 Subject: [pypy-dev] PyPy on Windows 7: Icon file for pypy.exe In-Reply-To: References: Message-ID: You can use rcedit : rcedit pypy.exe --set-icon pypy.ico Or use Resource Hacker (a GUI tool). Reminds me: I need to update the Chocolatey package for PyPy. I haven't touched it since 2.2... On Mon, May 4, 2015 at 3:37 AM, Armin Rigo wrote: > Hi Eun, > > On 4 May 2015 at 09:58, Eun Che wrote: > > I wished to contribute via Bitbucket fork, but i'm so new to that, don't > > know how to do. So I had to send this mail. > > Thanks! Can you give us a link that explains how to embed this icon > inside pypy.exe? Maybe it's just a command-line option of the linker > command, but I wouldn't be surprized if it's more involved. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Mon May 4 17:16:13 2015 From: arigo at tunes.org (Armin Rigo) Date: Mon, 4 May 2015 17:16:13 +0200 Subject: [pypy-dev] PyPy on Windows 7: Icon file for pypy.exe In-Reply-To: References: Message-ID: Hi all, On 4 May 2015 at 16:44, Ryan Gonzalez wrote: > You can use rcedit: I found the official way described here: http://doc.qt.io/qt-4.8/appicon.html . Interestingly, Windows looks like the simplest of all the platforms described there. The "rc" tool (unlike "rcedit") comes preinstalled together with Visual C++. A bient?t, Armin. From matti.picus at gmail.com Mon May 4 17:44:16 2015 From: matti.picus at gmail.com (Matti Picus) Date: Mon, 04 May 2015 18:44:16 +0300 Subject: [pypy-dev] PyPy on Windows 7: Icon file for pypy.exe In-Reply-To: References: Message-ID: <554793D0.1040006@gmail.com> An HTML attachment was scrubbed... URL: From yury at shurup.com Mon May 4 22:12:39 2015 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 04 May 2015 22:12:39 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: References: <201505032326.t43NQrRm019774@fido.openend.se> Message-ID: <1430770359.8013.42.camel@newpride> On Mon, 2015-05-04 at 06:06 -0500, Samuel Villamonte via pypy-dev wrote: > +1 here! May I suggest using Transifex? I'm not very excited about Transifex, ever since they shut down the open source version of their tool and now only offer a SaaS solution in as far as I know. How about Pootle? I've heard GNU project now has an instance if you don't want to host it yourselves... -- Sincerely yours, Yury V. Zaytsev From yury at shurup.com Mon May 4 22:21:45 2015 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 04 May 2015 22:21:45 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: <20150504032743.GV5663@ando.pearwood.info> References: <201505032326.t43NQrRm019774@fido.openend.se> <20150504032743.GV5663@ando.pearwood.info> Message-ID: <1430770905.8013.49.camel@newpride> On Mon, 2015-05-04 at 13:27 +1000, Steven D'Aprano wrote: > > so I think that the days when all programmers must learn English are > slowly fading away, just like the days when all mathematicians had to > learn German. I hope not; we can argue whether this language should be English or something else (e.g. Esperanto, or Latin, or ancient Greek for that matter), but IMO for *professional* programmers and mathematicians, there should be a common natural communication language. Still, don't get me wrong, I'm +1 on localization. It's going to be quite some effort, but it's well worth it. -- Sincerely yours, Yury V. Zaytsev From cfbolz at gmx.de Mon May 4 22:24:43 2015 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Mon, 04 May 2015 22:24:43 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: <1430770905.8013.49.camel@newpride> References: <201505032326.t43NQrRm019774@fido.openend.se> <20150504032743.GV5663@ando.pearwood.info> <1430770905.8013.49.camel@newpride> Message-ID: <5547D58B.4000509@gmx.de> On 04/05/15 22:21, Yury V. Zaytsev wrote: > On Mon, 2015-05-04 at 13:27 +1000, Steven D'Aprano wrote: >> >> so I think that the days when all programmers must learn English are >> slowly fading away, just like the days when all mathematicians had to >> learn German. > > I hope not; we can argue whether this language should be English or > something else (e.g. Esperanto, or Latin, or ancient Greek for that > matter), but IMO for *professional* programmers and mathematicians, > there should be a common natural communication language. Yes, well. It's precisely *not* about professional programmers in this issue. Cheers, CF From yury at shurup.com Mon May 4 22:26:27 2015 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 04 May 2015 22:26:27 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: <5547D58B.4000509@gmx.de> References: <201505032326.t43NQrRm019774@fido.openend.se> <20150504032743.GV5663@ando.pearwood.info> <1430770905.8013.49.camel@newpride> <5547D58B.4000509@gmx.de> Message-ID: <1430771187.8013.50.camel@newpride> On Mon, 2015-05-04 at 22:24 +0200, Carl Friedrich Bolz wrote: > > Yes, well. It's precisely *not* about professional programmers in this > issue. That's why, as I said, I'm totally +1 on it ;-) -- Sincerely yours, Yury V. Zaytsev From santagada at gmail.com Tue May 5 01:55:35 2015 From: santagada at gmail.com (Leonardo Santagada) Date: Mon, 4 May 2015 20:55:35 -0300 Subject: [pypy-dev] (no subject) In-Reply-To: <1430771187.8013.50.camel@newpride> References: <201505032326.t43NQrRm019774@fido.openend.se> <20150504032743.GV5663@ando.pearwood.info> <1430770905.8013.49.camel@newpride> <5547D58B.4000509@gmx.de> <1430771187.8013.50.camel@newpride> Message-ID: As we are transitioning to internationalized messages should/could we also get more descritive error messages? It could always be just a phrase and a link, eg: instead of TypeError: x() takes no arguments (1 given) we output: TypeError: x() takes no arguments (1 given) (maybe you forgot the self attribute (https://docs.python.org/2/tutorial/classes.html#random-remarks)) I think for this to move forward someone (me) would need to write a pep about it, I just want to know if you think it is a good idea, or there is an obvious problem that I'm not seeing. On Mon, May 4, 2015 at 5:26 PM, Yury V. Zaytsev wrote: > On Mon, 2015-05-04 at 22:24 +0200, Carl Friedrich Bolz wrote: > > > > Yes, well. It's precisely *not* about professional programmers in this > > issue. > > That's why, as I said, I'm totally +1 on it ;-) > > -- > Sincerely yours, > Yury V. Zaytsev > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Leonardo Santagada -------------- next part -------------- An HTML attachment was scrubbed... URL: From reingart at gmail.com Tue May 5 09:57:01 2015 From: reingart at gmail.com (Mariano Reingart) Date: Tue, 5 May 2015 04:57:01 -0300 Subject: [pypy-dev] Internationalization Proposal (was Re: (no subject)) Message-ID: Hello, and sorry for being late to the discussion: Thanks Laura for bringing this up, and thanks everyone else for the interest. As I wasn't subscribed to this list early, I didn't get a copy of the emails so I cannot answer each one in-line, but I'll do my best trying to reconstruct the thread citing the original messages (and fixing the subject line). Sorry for the very long mail, but I wanted to address all the concerns and topics. On Mon May 4 05:27:44 CEST 2015, Steven D'Aprano wrote: I've just added this comment to the issue on the bug tracker: http://bugs.python.org/issue16344 Thanks, I updated the issue pointing a new CPython clone in GitHub were I'll try to work reorganizing and drafting the PEP https://github.com/reingart/cpython I'll try to clone PyPy too, I think it could be even easier to internationalize, as it is pure python and no C API should be changed, but please correct me if I'm wrong. Anyone can write a PEP and have it sit on their computer's hard drive. > Unless he follows the process for submitting it for discussion, it might > as well not exist. That's not exactly true, I'm working in the PEP publicly since 2010 (it was first uploaded in the local users group): http://python.org.ar/wiki/TracebackInternationalizationProposal It not only was discussed in local and international mailing lists, it also was sent to python-ideas as PEP 1 indicates: https://mail.python.org/pipermail/python-ideas/2012-October/017493.html (seems there aren't responses anyway) Also, the draft PEP was attached to the issue (and was completely ignored by the core dev that closed and rejected the proposal, not even answering other core dev that pointed that the PEP was there) What I didn't do is sending it to PEP editors, my fault (I confess I didn't reached that part of PEP1 that time), as I thought it would be rejected anyway: reception was mixed (I didn't find other core dev willing to strong "champion" it -but re reading the PEP1 maybe I could champion it myself-) and why it would have a better luck, if the issue was rejected without any further discussion?. Looking at the volume of comments on the issue at the tracker link > above, I don't think Mariano has been *ignored*. Possibly the > conservativism of Python-Dev has struck again. There was a lot of > resistance to the idea of allowing non-ASCII identifiers when that was > first proposed too. Well, it depends on the perspective (see above), and if you see at the related blocking issue, it was really ignored (no new comments after my patch trying to fix it): http://bugs.python.org/issue9769 PEP 1 also states "wherever possible, long open-ended discussions on public mailing lists should be avoided", so with the previous facts, I did back off for a better opportunity. The same year in November, I spoke with some core devs that came to PyCon Argentina 2012 (that BTW I was the chair and tried to organize a CPython core sprint to work on this topics), specifically Brett Cannon, but he didn't have experience on internationalization. This year I thought the GSoC was an alternative, but again, I was wrong (at least I didn't received any feedback about why it was not accepted). I could continue but I'll not bother you. Note: I am not blaming anyone, just trying to give a summary of my experience. My own feeling is that CPython should not internationalise error > messages *yet*. But I would like to see a third-party implementation, > possibly something that can be switched on and off at the interactive > interpreter, or in iPython, and see how useful it is. But that's > something which can be discussed in the PEP. Why not yet? Anyway, as far as I remember, a 3rd party implementation is not viable: messages are already formatted when they reach exception hook, and the hook could not always be honored, beside that internal cpython message formatting seems to lack any UTF-8 nor internationalization support (positional parameters reorder, pluralization forms, etc.), just to mention technical issues I remember. And yes, the proposal contemplates turning translation on and off, even at runtime. How would the usefulness be evaluated? (field research, surveys, how many, where, etc.) How was it done in previous PEPs? (for example, what was done for other peps, like pep-3131) I really want to understand the process. I think that if Mariano had a proof of concept, and evidence that it > would be useful in practice as well as theory, then I think the PSF > should be asked to consider funding this as a diversity measure. The proof of concept was already there, you can see the original proposed patch: http://bugs.python.org/file27756/traceback_internationalization_proposal.patch In fact it is a half-backed prototype (the C part was there, translation was implicit, but the python part would need some more work, as translation would have to be more explicit, and messages collection tools needs to be developed). It surely has some other aspect to polish. The problem seems to be the evidence, that's why I proposed this as a GSoC project: to be able to develop a pilot implementation and get more facts after testing it and doing some research, as the effort is not trivial. On Mon May 4 10:00:49 CEST 2015, Armin Rigo arigo at tunes.org wrote: > I'm absolutely +1 on the idea. [snip] Thanks, and I agree with everything you said Even "syntax error!" is fine if your programming language is BASIC but the first difference with BASIC is that we can get commonly > many more error messages. Well, recently I did a little test with my 8 years old kid, starting with the python turtle module. He had difficulties even with the pronunciation of some error messages, and he has always to ask for the translation & help (until he gave up). I have observed the same behavior with even older students. Translation is not the only culprit, but I think it is a start (there are some educational theories that could be used to explain and overcome this, but I don't want to make this any longer). > I think that allowing them to be translated is a good first step. Exactly, I like the PostgreSQL motto: *PostgreSQL programs (server and client) can issue their messages in your favorite language ? if the messages have been translated. Creating and maintaining translated message sets needs the help of people who speak their own language well and want to contribute to the PostgreSQL effort. You do not have to be a programmer at all to do this. * *...* *We won't judge your language skills ? this section is about software tools.* http://www.postgresql.org/docs/9.4/static/nls-translator.html On Mon May 4 22:21:45 CEST 2015, Yury V. Zaytsev yury at shurup.com wrote: > > > > so I think that the days when all programmers must learn English are > > slowly fading away, just like the days when all mathematicians had to > > learn German. I hope not; we can argue whether this language should be English or > something else (e.g. Esperanto, or Latin, or ancient Greek for that > matter), but IMO for *professional* programmers and mathematicians, > there should be a common natural communication language. > Please define what a professional programmer is... And please define what a "learn English" means... For non native speakers, it would mean formally a "intermediate to proficiency English level" (TOEFL, Cambridge First Certificate FCE, etc.), and we have to take that tests periodically (once each couple of years to not "lose" the skills), with a minimal preparation of around 8 weeks (considering you have a good English background, something that usually takes around 4-7 years of extracurricular courses & practice) As far as I know, at least in Argentina, that "English proficiency" doesn't seems to be a formal requisite to be a "professional" programmer (if that equates to a major degree in computer sciences or similar, not to mention younger students or non-technical users). In fact, nor "English" (nor any other foreign language) is specified in the ministerial resolution that regulates the careers in computer sciences, informatics, system analysis and software engineering (according the core basic curriculum recommendations): http://www.cs.uns.edu.ar/downloads/acreditacion/res786.09.pdf That is indirectly based on the Computing Curricula of ACM-IEEE, this seems a updated version: https://www.acm.org/education/CS2013-final-report.pdf Again, note that English seems not to be a core competency/skill (at least in a first sight). In fact it is only mentioned as an example of a natural language, and it is only required for some courses given in English. Of course, that's doesn't mean that sooner or later students will be exposed to some English courses and materials. In fact, most undergraduate programs will include "starter" English level courses in Argentina, but they could be suspended or taken in parallel to Programming I, our equivalent to CS101 (and no, English is usually not "correlative" to technical courses, just complementary). For example, after a quick search, see this computer information systems engineering degree (from one of the most renowned Universities in Latin America), where English is optional, and you can even choose other foreign languages (including German, French, Italian, Portuguese) http://www.uba.ar/download/academicos/carreras/ingenieriainformatica.pdf This is the updated plan that even states "only it will considered one foreign language" (and BTW, just 4 credits = 4 hours per week per one semester, if you can learn a foreign language in just 64 hours...): http://www.fi.uba.ar/archivos/Actualizacion%202011%20Plan%20de%20Estudios%20Informatica.pdf You can imagine that this only get worse in the elementary and secondary schools (or even in other university degrees not related to IT), although English courses are being included gradually in the last years in all the educational levels. As a personal comment, so far in my academic career, I only faced an intermediate English knowledge (and certification) requirement for applying to a Ph.D. to study in the EU or US. I hold a Bachelor of Computer Analysis, and a Master in Free Software and Open Source (online from Spain), and I'm finishing a Bachelors in Education without any such certification nor requirement (the last ones didn't even have English courses). BTW, for example, if I study for a Computer Science Ph.D. in Argentina, the thesis should be written in Spanish (only exceptionally it is allowed to be written in English): http://postgrado.info.unlp.edu.ar/La_Institucion/Reglamento.pdf The funny thing if that for foreign students, Spanish language examination could be also a requisite, as if I go to Brazil, I could be required to learn Portuguese, etc. As a side note, another point is that we, as teachers, cannot even teach legally English courses if we're not specially certified for that (not only passed a Proficiency/Translator exam, also there are special teacher formation requirements) In summary, assuming English is a requisite to be a *professional* programmer is wrong and impractical IMHO (of course, YMMV). Still, don't get me wrong, I'm +1 on localization. It's going to be > quite some effort, but it's well worth it. Great! And please don't get me wrong too, I only want to Python be more diverse and easy for non English speakers. And yes, it will be a great effort ;-) BTW, the effort is already happening in other areas/aspects, for example, see the upcoming SciPy Latin America, that aims to mix English, Portuguese and Spanish: http://conf.scipyla.org/event/press Other conferences offered live translation to Spanish, for keynote English talks: http://ar.pycon.org/pyconar2014/ I think that proves that knowledge can be communicated in several languages, and I bet you'll have a bigger audience & more feedback if you speak in Portu?ol (Portuguese + Spanish), than in English, at least in the Southern Cone. On Mon, 2015-05-04 at 22:24 +0200, Carl Friedrich Bolz wrote: > > Yes, well. It's precisely *not* about professional programmers in this > issue. Sorry to insist, but please this are the exact phrases that are a bit embarrassing to many of us. Are those people (mentioned early) that gave Python talks in Spanish or Portuguese not professional programmers? I even already write programs with identifiers and exceptions in Spanish, what's wrong with that? BTW, we don't have even right English translations for some words in Spanish (and this is just a project for electronic invoices) https://github.com/reingart/pyafipws Forcing to translate every term and write programs exclusively in English not only is harder, many times brings just more confusion, even the official government webservices' documentation (AFIP = Argentina's IRS) uses just Get/Set and very general terms in English, maybe due a Java getter/setter legacy, and almost all the complex data types, methods and messages are Spanish (and no, this is for programmers, not for final users): http://www.afip.gov.ar/fe/documentos/manua_desarrolladorCOMPGv25.pdf And just to clarify, CPython could be still developed in English, no translation will ever touch a source code file. On Mon, May 4, 2015 at 8:55 PM, Leonardo Santagada wrote: > As we are transitioning to internationalized messages should/could we also > get more descritive error messages? It could always be just a phrase and a > link, eg: > instead of > TypeError: x() takes no arguments (1 given) > we output: > TypeError: x() takes no arguments (1 given) (maybe you forgot the self > attribute (https://docs.python.org/2/tutorial/classes.html#random-remarks > )) > Yes, this was also discussed, but I think it is a second step, and internationalization can be useful here too due diversity, to clear ambiguities or fix to rewrite complex messages, as different languages has different writing styles, rules, viewpoints, etc. Best regards, Mariano Reingart http://www.sistemasagiles.com.ar http://reingart.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue May 5 10:23:39 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 5 May 2015 10:23:39 +0200 Subject: [pypy-dev] Internationalization Proposal (was Re: (no subject)) In-Reply-To: References: Message-ID: Hi Mariano. Welcome on board! As expressed before, we would be happy to have internationalization in PyPy as I would like to have a discussion about which of the tools are suitable for driving that forward. That said, this is a list of development of PyPy and it's not a list to discuss how python-dev handles issues. I'm more than happy to help with any constructive work w.r.t. internationalization and discussing what went wrong on the cpython bug tracker is not one of those. It's also quite hard to read mails that answer all of the before mails in one go - for example I'm very much interested in seeing PyPy work go forward and I'm very much not interested in the experience of trying to get python-dev attention onto any topic so if you can, please respond one-mail-per-topic in the future, thanks! Cheers, fijal On Tue, May 5, 2015 at 9:57 AM, Mariano Reingart wrote: > Hello, and sorry for being late to the discussion: > > Thanks Laura for bringing this up, and thanks everyone else for the > interest. > > As I wasn't subscribed to this list early, I didn't get a copy of the emails > so I cannot answer each one in-line, but I'll do my best trying to > reconstruct the thread citing the original messages (and fixing the subject > line). > > Sorry for the very long mail, but I wanted to address all the concerns and > topics. > > > > On Mon May 4 05:27:44 CEST 2015, Steven D'Aprano wrote: > >> I've just added this comment to the issue on the bug tracker: >> >> >> >> http://bugs.python.org/issue16344 > > > > Thanks, I updated the issue pointing a new CPython clone in GitHub were I'll > try to work reorganizing and drafting the PEP > > https://github.com/reingart/cpython > > I'll try to clone PyPy too, I think it could be even easier to > internationalize, as it is pure python and no C API should be changed, but > please correct me if I'm wrong. > >> Anyone can write a PEP and have it sit on their computer's hard drive. >> Unless he follows the process for submitting it for discussion, it might >> as well not exist. > > > That's not exactly true, I'm working in the PEP publicly since 2010 (it was > first uploaded in the local users group): > > http://python.org.ar/wiki/TracebackInternationalizationProposal > > It not only was discussed in local and international mailing lists, it also > was sent to python-ideas as PEP 1 indicates: > > https://mail.python.org/pipermail/python-ideas/2012-October/017493.html > > (seems there aren't responses anyway) > > Also, the draft PEP was attached to the issue (and was completely ignored by > the core dev that closed and rejected the proposal, not even answering other > core dev that pointed that the PEP was there) > > What I didn't do is sending it to PEP editors, my fault (I confess I didn't > reached that part of PEP1 that time), as I thought it would be rejected > anyway: reception was mixed (I didn't find other core dev willing to strong > "champion" it -but re reading the PEP1 maybe I could champion it myself-) > and why it would have a better luck, if the issue was rejected without any > further discussion?. > >> Looking at the volume of comments on the issue at the tracker link >> above, I don't think Mariano has been *ignored*. Possibly the >> conservativism of Python-Dev has struck again. There was a lot of >> resistance to the idea of allowing non-ASCII identifiers when that was >> first proposed too. > > > Well, it depends on the perspective (see above), and if you see at the > related blocking issue, it was really ignored (no new comments after my > patch trying to fix it): > > http://bugs.python.org/issue9769 > > PEP 1 also states "wherever possible, long open-ended discussions on public > mailing lists should be avoided", so with the previous facts, I did back off > for a better opportunity. > > The same year in November, I spoke with some core devs that came to PyCon > Argentina 2012 (that BTW I was the chair and tried to organize a CPython > core sprint to work on this topics), specifically Brett Cannon, but he > didn't have experience on internationalization. > > This year I thought the GSoC was an alternative, but again, I was wrong (at > least I didn't received any feedback about why it was not accepted). > > I could continue but I'll not bother you. > > Note: I am not blaming anyone, just trying to give a summary of my > experience. > >> My own feeling is that CPython should not internationalise error >> messages *yet*. But I would like to see a third-party implementation, >> possibly something that can be switched on and off at the interactive >> interpreter, or in iPython, and see how useful it is. But that's >> something which can be discussed in the PEP. > > > Why not yet? > > Anyway, as far as I remember, a 3rd party implementation is not viable: > messages are already formatted when they reach exception hook, and the hook > could not always be honored, beside that internal cpython message formatting > seems to lack any UTF-8 nor internationalization support (positional > parameters reorder, pluralization forms, etc.), just to mention technical > issues I remember. > > And yes, the proposal contemplates turning translation on and off, even at > runtime. > > How would the usefulness be evaluated? > (field research, surveys, how many, where, etc.) > How was it done in previous PEPs? > (for example, what was done for other peps, like pep-3131) > > I really want to understand the process. > >> I think that if Mariano had a proof of concept, and evidence that it >> would be useful in practice as well as theory, then I think the PSF >> should be asked to consider funding this as a diversity measure. > > > The proof of concept was already there, you can see the original proposed > patch: > > http://bugs.python.org/file27756/traceback_internationalization_proposal.patch > > In fact it is a half-backed prototype (the C part was there, translation was > implicit, but the python part would need some more work, as translation > would have to be more explicit, and messages collection tools needs to be > developed). > It surely has some other aspect to polish. > > The problem seems to be the evidence, that's why I proposed this as a GSoC > project: to be able to develop a pilot implementation and get more facts > after testing it and doing some research, as the effort is not trivial. > > > > On Mon May 4 10:00:49 CEST 2015, Armin Rigo arigo at tunes.org wrote: >> >> I'm absolutely +1 on the idea. >> >> [snip] > > > Thanks, and I agree with everything you said > >> Even "syntax error!" is fine if your programming language is BASIC >> >> but the first difference with BASIC is that we can get commonly >> many more error messages. > > > Well, recently I did a little test with my 8 years old kid, starting with > the python turtle module. > He had difficulties even with the pronunciation of some error messages, and > he has always to ask for the translation & help (until he gave up). > I have observed the same behavior with even older students. > Translation is not the only culprit, but I think it is a start (there are > some educational theories that could be used to explain and overcome this, > but I don't want to make this any longer). > >> >> I think that allowing them to be translated is a good first step. > > > Exactly, I like the PostgreSQL motto: > > PostgreSQL programs (server and client) can issue their messages in your > favorite language ? if the messages have been translated. Creating and > maintaining translated message sets needs the help of people who speak their > own language well and want to contribute to the PostgreSQL effort. You do > not have to be a programmer at all to do this. > ... > We won't judge your language skills ? this section is about software tools. > > > http://www.postgresql.org/docs/9.4/static/nls-translator.html > > > > On Mon May 4 22:21:45 CEST 2015, Yury V. Zaytsev yury at shurup.com wrote: >> >> > >> > so I think that the days when all programmers must learn English are >> > slowly fading away, just like the days when all mathematicians had to >> > learn German. >> >> I hope not; we can argue whether this language should be English or >> something else (e.g. Esperanto, or Latin, or ancient Greek for that >> matter), but IMO for *professional* programmers and mathematicians, >> there should be a common natural communication language. > > > Please define what a professional programmer is... > And please define what a "learn English" means... > > For non native speakers, it would mean formally a "intermediate to > proficiency English level" (TOEFL, Cambridge First Certificate FCE, etc.), > and we have to take that tests periodically (once each couple of years to > not "lose" the skills), with a minimal preparation of around 8 weeks > (considering you have a good English background, something that usually > takes around 4-7 years of extracurricular courses & practice) > > As far as I know, at least in Argentina, that "English proficiency" doesn't > seems to be a formal requisite to be a "professional" programmer (if that > equates to a major degree in computer sciences or similar, not to mention > younger students or non-technical users). > In fact, nor "English" (nor any other foreign language) is specified in the > ministerial resolution that regulates the careers in computer sciences, > informatics, system analysis and software engineering (according the core > basic curriculum recommendations): > > http://www.cs.uns.edu.ar/downloads/acreditacion/res786.09.pdf > > That is indirectly based on the Computing Curricula of ACM-IEEE, this seems > a updated version: > > https://www.acm.org/education/CS2013-final-report.pdf > > Again, note that English seems not to be a core competency/skill (at least > in a first sight). > In fact it is only mentioned as an example of a natural language, and it is > only required for some courses given in English. > > Of course, that's doesn't mean that sooner or later students will be exposed > to some English courses and materials. > In fact, most undergraduate programs will include "starter" English level > courses in Argentina, but they could be suspended or taken in parallel to > Programming I, our equivalent to CS101 (and no, English is usually not > "correlative" to technical courses, just complementary). > For example, after a quick search, see this computer information systems > engineering degree (from one of the most renowned Universities in Latin > America), where English is optional, and you can even choose other foreign > languages (including German, French, Italian, Portuguese) > > http://www.uba.ar/download/academicos/carreras/ingenieriainformatica.pdf > > This is the updated plan that even states "only it will considered one > foreign language" (and BTW, just 4 credits = 4 hours per week per one > semester, if you can learn a foreign language in just 64 hours...): > > http://www.fi.uba.ar/archivos/Actualizacion%202011%20Plan%20de%20Estudios%20Informatica.pdf > > You can imagine that this only get worse in the elementary and secondary > schools (or even in other university degrees not related to IT), although > English courses are being included gradually in the last years in all the > educational levels. > > As a personal comment, so far in my academic career, I only faced an > intermediate English knowledge (and certification) requirement for applying > to a Ph.D. to study in the EU or US. > I hold a Bachelor of Computer Analysis, and a Master in Free Software and > Open Source (online from Spain), and I'm finishing a Bachelors in Education > without any such certification nor requirement (the last ones didn't even > have English courses). > BTW, for example, if I study for a Computer Science Ph.D. in Argentina, the > thesis should be written in Spanish (only exceptionally it is allowed to be > written in English): > > http://postgrado.info.unlp.edu.ar/La_Institucion/Reglamento.pdf > > The funny thing if that for foreign students, Spanish language examination > could be also a requisite, as if I go to Brazil, I could be required to > learn Portuguese, etc. > > As a side note, another point is that we, as teachers, cannot even teach > legally English courses if we're not specially certified for that (not only > passed a Proficiency/Translator exam, also there are special teacher > formation requirements) > > In summary, assuming English is a requisite to be a *professional* > programmer is wrong and impractical IMHO (of course, YMMV). > >> Still, don't get me wrong, I'm +1 on localization. It's going to be >> quite some effort, but it's well worth it. > > > Great! > And please don't get me wrong too, I only want to Python be more diverse and > easy for non English speakers. > > And yes, it will be a great effort ;-) > > BTW, the effort is already happening in other areas/aspects, for example, > see the upcoming SciPy Latin America, that aims to mix English, Portuguese > and Spanish: > > http://conf.scipyla.org/event/press > > Other conferences offered live translation to Spanish, for keynote English > talks: > > http://ar.pycon.org/pyconar2014/ > > I think that proves that knowledge can be communicated in several languages, > and I bet you'll have a bigger audience & more feedback if you speak in > Portu?ol (Portuguese + Spanish), than in English, at least in the Southern > Cone. > > > > On Mon, 2015-05-04 at 22:24 +0200, Carl Friedrich Bolz wrote: >> >> Yes, well. It's precisely *not* about professional programmers in this >> issue. > > Sorry to insist, but please this are the exact phrases that are a bit > embarrassing to many of us. > > Are those people (mentioned early) that gave Python talks in Spanish or > Portuguese not professional programmers? > > I even already write programs with identifiers and exceptions in Spanish, > what's wrong with that? > BTW, we don't have even right English translations for some words in Spanish > (and this is just a project for electronic invoices) > > https://github.com/reingart/pyafipws > > Forcing to translate every term and write programs exclusively in English > not only is harder, many times brings just more confusion, even the official > government webservices' documentation (AFIP = Argentina's IRS) uses just > Get/Set and very general terms in English, maybe due a Java getter/setter > legacy, and almost all the complex data types, methods and messages are > Spanish (and no, this is for programmers, not for final users): > > http://www.afip.gov.ar/fe/documentos/manua_desarrolladorCOMPGv25.pdf > > > And just to clarify, CPython could be still developed in English, no > translation will ever touch a source code file. > > > > On Mon, May 4, 2015 at 8:55 PM, Leonardo Santagada > wrote: >> >> As we are transitioning to internationalized messages should/could we also >> get more descritive error messages? It could always be just a phrase and a >> link, eg: >> instead of >> TypeError: x() takes no arguments (1 given) >> we output: >> TypeError: x() takes no arguments (1 given) (maybe you forgot the self >> attribute (https://docs.python.org/2/tutorial/classes.html#random-remarks)) > > > Yes, this was also discussed, but I think it is a second step, and > internationalization can be useful here too due diversity, to clear > ambiguities or fix to rewrite complex messages, as different languages has > different writing styles, rules, viewpoints, etc. > > > > Best regards, > > > Mariano Reingart > http://www.sistemasagiles.com.ar > http://reingart.blogspot.com > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Tue May 5 10:31:49 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 5 May 2015 10:31:49 +0200 Subject: [pypy-dev] (no subject) In-Reply-To: References: <201505032326.t43NQrRm019774@fido.openend.se> <20150504032743.GV5663@ando.pearwood.info> <1430770905.8013.49.camel@newpride> <5547D58B.4000509@gmx.de> <1430771187.8013.50.camel@newpride> Message-ID: 2015-05-05 1:55 GMT+02:00 Leonardo Santagada : > As we are transitioning to internationalized messages should/could we also > get more descritive error messages? It could always be just a phrase and a > link, eg: > instead of > TypeError: x() takes no arguments (1 given) > we output: > TypeError: x() takes no arguments (1 given) (maybe you forgot the self > attribute (https://docs.python.org/2/tutorial/classes.html#random-remarks > )) > > I think for this to move forward someone (me) would need to write a pep > about it, I just want to know if you think it is a good idea, or there is > an obvious problem that I'm not seeing. > This could be implemented as a custom language ("en-DESCRIPTIVE") so that you are free to experiment without changing any code... > > > On Mon, May 4, 2015 at 5:26 PM, Yury V. Zaytsev wrote: > >> On Mon, 2015-05-04 at 22:24 +0200, Carl Friedrich Bolz wrote: >> > >> > Yes, well. It's precisely *not* about professional programmers in this >> > issue. >> >> That's why, as I said, I'm totally +1 on it ;-) >> >> -- >> Sincerely yours, >> Yury V. Zaytsev >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> > > > > -- > > Leonardo Santagada > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue May 5 14:41:52 2015 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 May 2015 14:41:52 +0200 Subject: [pypy-dev] Internationalization Proposal (was Re: (no subject)) In-Reply-To: References: Message-ID: Hi Mariano, On 5 May 2015 at 09:57, Mariano Reingart wrote: > I'll try to clone PyPy too, I think it could be even easier to > internationalize, as it is pure python and no C API should be changed, but > please correct me if I'm wrong. It's easier in some ways, but harder in others, because there is no experience about doing that on an RPython project (unlike C). To get started: * let's say the goal is to use gettext via the dgettext() API. This is a function that takes and returns a C string, i.e. a "char *", which is different from an RPython string. We need to interface to dgettext() via rffi (see examples in rpython/rlib/*.py, like e.g. zrlib.py or rpoll.py or others). * I think the sanest is to assume that the language doesn't change dynamically, so that we can use caching techniques; for now it can be done by adding "@jit.elidable" as a decorator, which mean that at least the JIT will constant-fold calls to dgettext(). * the first place to use this would be in the error message construction. Unfortunately, this is also a place where we do custom things instead of relying on a standard printf() format: pypy/interpreter/error.py. To see examples, grep for "raise oefmt" anywhere in pypy/. A message is specified like "expected %d arguments, got %d", but it is broken in advance (during "translation", i.e. when we turn PyPy from Python code to an actual executable) into two strings, "expected " and " arguments, got "; at runtime we build the final string by concatenating the pieces. This logic needs to change, but the idea of needing only to concatenate some pieces should stay, as it is essential to get good JIT performance. We probably need to call dgettext() on the whole input string, and then split it up to prepare the concatenation-of-pieces for future calls --- done at runtime but still cached so that it occurs only once for each input string. (Obviously we need a way to reorder the arguments, too.) This should be discussed in more details. Maybe you should show up on irc, channel #pypy on irc.freenode.net. A bient?t, Armin. From reingart at gmail.com Tue May 5 19:03:53 2015 From: reingart at gmail.com (Mariano Reingart) Date: Tue, 5 May 2015 14:03:53 -0300 Subject: [pypy-dev] Internationalization Proposal (was Re: (no subject)) In-Reply-To: References: Message-ID: On Tue, May 5, 2015 at 5:23 AM, Maciej Fijalkowski wrote: > Hi Mariano. > > Welcome on board! > > As expressed before, we would be happy to have internationalization in > PyPy as I would like to have a discussion about which of the tools are > suitable for driving that forward. > > Well, in the draft PEP is explained: gettext Beside tools, this would require a lot of coordination and work so the translations get coherence, useful reviews, good translation memory, etc. Although the GSoC proposal was for CPython, many things could be relevant to PyPy, including: backwards compatibility, development, performance, packaging, maintenance, effort estimation, consistency and other internationalization "difficulties". That said, this is a list of development of PyPy and it's not a list > to discuss how python-dev handles issues. I'm more than happy to help > with any constructive work w.r.t. internationalization and discussing > what went wrong on the cpython bug tracker is not one of those. > > Sorry, that was not my intention. I only wanted to bring some clarifications about the precedents of my internationalization proposal, that by the way, were raised here by other PyPy devs, not by me. Anyway, I think that a 5 year old experience could be useful, where many other python devs already expressed concerns, proposed tools or had supportive comments / suggestions. > It's also quite hard to read mails that answer all of the before mails > in one go - for example I'm very much interested in seeing PyPy work > go forward and I'm very much not interested in the experience of > trying to get python-dev attention onto any topic so if you can, > please respond one-mail-per-topic in the future, thanks! > > Yes, I already apologized for answering a long mail, I didn't was subscribed to this list so I couldn't answer each one individually. You're right about the python-dev, and in fact I had no plans in the short term to insist there. Also I think this goes further than that, and could be useful to to the whole python community (not only pypy or cpython, but also for other python implementations and projects). Best regards, Mariano Reingart http://www.sistemasagiles.com.ar http://reingart.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From reingart at gmail.com Tue May 5 20:21:46 2015 From: reingart at gmail.com (Mariano Reingart) Date: Tue, 5 May 2015 15:21:46 -0300 Subject: [pypy-dev] Internationalization Proposal (was Re: (no subject)) In-Reply-To: References: Message-ID: On Tue, May 5, 2015 at 9:41 AM, Armin Rigo wrote: > Hi Mariano, > > On 5 May 2015 at 09:57, Mariano Reingart wrote: > > I'll try to clone PyPy too, I think it could be even easier to > > internationalize, as it is pure python and no C API should be changed, > but > > please correct me if I'm wrong. > > It's easier in some ways, but harder in others, because there is no > experience about doing that on an RPython project (unlike C). To get > started: > > * let's say the goal is to use gettext via the dgettext() API. This > is a function that takes and returns a C string, i.e. a "char *", > which is different from an RPython string. We need to interface to > dgettext() via rffi (see examples in rpython/rlib/*.py, like e.g. > zrlib.py or rpoll.py or others). > > Or using a pure python implementation already in the stdlib? https://hg.python.org/cpython/file/2.7/Lib/gettext.py > * I think the sanest is to assume that the language doesn't change > dynamically, so that we can use caching techniques; for now it can be > done by adding "@jit.elidable" as a decorator, which mean that at > least the JIT will constant-fold calls to dgettext(). > Ok, I'll investigate this... > * the first place to use this would be in the error message > construction. Unfortunately, this is also a place where we do custom > things instead of relying on a standard printf() format: > pypy/interpreter/error.py. To see examples, grep for "raise oefmt" > anywhere in pypy/. A message is specified like "expected %d > arguments, got %d", but it is broken in advance (during "translation", > i.e. when we turn PyPy from Python code to an actual executable) into > two strings, "expected " and " arguments, got "; at runtime we build > the final string by concatenating the pieces. This logic needs to > change, but the idea of needing only to concatenate some pieces should > stay, as it is essential to get good JIT performance. We probably > need to call dgettext() on the whole input string, and then split it > up to prepare the concatenation-of-pieces for future calls --- done at > runtime but still cached so that it occurs only once for each input > string. (Obviously we need a way to reorder the arguments, too.) > Ok, understood. Anyway, do you think this really has any performance implication? dgettext is mainly a lookup and interpolation, cannot it just be used to return the final string? Splitting the text can be direct for some languages, but I don't know how it will work for left-to-right or non latin1 character sets... This should be discussed in more details. Maybe you should show up on > irc, channel #pypy on irc.freenode.net. > Ok, I'm downloading pypy and will build it and try to further investigate. BTW, I see a lot of heads in the mercurial repo, which version should I use? tip? I prefer communication using mailing list as I gives more time to properly reply, but you can tell me in the next week when will you be available and I'll try to show up in the IRC. Best regards, Mariano Reingart http://www.sistemasagiles.com.ar http://reingart.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue May 5 21:18:22 2015 From: arigo at tunes.org (Armin Rigo) Date: Tue, 5 May 2015 21:18:22 +0200 Subject: [pypy-dev] Internationalization Proposal (was Re: (no subject)) In-Reply-To: References: Message-ID: Hi Mariano, On 5 May 2015 at 20:21, Mariano Reingart wrote: > Or using a pure python implementation already in the stdlib? > > https://hg.python.org/cpython/file/2.7/Lib/gettext.py Ah, yes. This comes with potential performance issues, but let's not focus on performance to start with, as it is easily changed later. So it can be done by writing, in RPython, code like that---this is hackish and not terribly fast, but it should work, running the code within the multi-line quoted string as regular Python: w_translated_string = space.appexec([w_string], """(message): import gettext return gettext.dgettext('python', message) """) You can get started by adding this to the class OpErrFmtNoArgs in pypy/interpreter/error.py, to get translations for the calls ``raise oefmt(..., "string")`` that have no ``%`` argument at all: def get_w_value(self, space): w_value = self._w_value if w_value is None: # here 'self._value' is a string; turn it into an app-level string object w_value = space.wrap(self._value) # translate it! w_value = space.appexec([w_value], ...) self._w_value = w_value return w_value It can be tried (and tested!) without doing any compilation. Try it out by running pypy/bin/pyinteractive.py --withmod-struct -S and trying an example like typing ``float(None)``. This error message is in objspace/std/floatobject.py built with oefmt() and no ``%``, so should go via gettext now. (Be patient; it is running the complete PyPy on top of CPython.) When you get the basics started, write it as unit tests rather than just trying them out interactively: see for example pypy/interpreter/test/test_argument.py, class AppTestArgument, which tries to get random exceptions and checks the message. >> * I think the sanest is to assume that the language doesn't change >> dynamically, so that we can use caching techniques; for now it can be >> done by adding "@jit.elidable" as a decorator, which mean that at >> least the JIT will constant-fold calls to dgettext(). > > Ok, I'll investigate this... This is also "a performance issue for later". :-) > dgettext is mainly a lookup and interpolation, cannot it just be used to > return the final string? Yes, of course we cannot use gettext to translate parts of the string --- this is not going to work, translation-wise. The issue I discuss here is purely technical: the string "got more than %d args" is written like that in the RPython source, but actually split and stored in the pypy binary as two strings, the left and right parts. We need to fix this by doing this at *runtime* instead: from the full string "got more than %d args", we'd invoke gettext and go to its translation (possibly using "%$2d" or some other convention to reorder the parameters); then split the result; and finally cache these pieces of strings and the reordering of the arguments. The actual splitting is only an optimization, but it needs to be done once at runtime instead of being done earlier. > BTW, I see a lot of heads in the mercurial repo, which version should I use? > tip? Yes, and make a new branch if you want to commit. > I prefer communication using mailing list as I gives more time to properly > reply That's fine! Armin From fijall at gmail.com Wed May 6 23:15:04 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 6 May 2015 23:15:04 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: The gdbm library is not thread-safe. Add a global lock. In-Reply-To: <20150506193916.136A61C0FE0@cobra.cs.uni-duesseldorf.de> References: <20150506193916.136A61C0FE0@cobra.cs.uni-duesseldorf.de> Message-ID: I was thinking, maybe instead we can add a feature to cffi "don't release the GIL" and use that there? (it would be faster for example) On Wed, May 6, 2015 at 9:39 PM, arigo wrote: > Author: Armin Rigo > Branch: > Changeset: r77171:df44050e8e33 > Date: 2015-05-06 21:39 +0200 > http://bitbucket.org/pypy/pypy/changeset/df44050e8e33/ > > Log: The gdbm library is not thread-safe. Add a global lock. > > diff --git a/lib_pypy/gdbm.py b/lib_pypy/gdbm.py > --- a/lib_pypy/gdbm.py > +++ b/lib_pypy/gdbm.py > @@ -1,4 +1,6 @@ > import cffi, os, sys > +import thread > +_lock = thread.allocate_lock() > > ffi = cffi.FFI() > ffi.cdef(''' > @@ -40,6 +42,7 @@ > > try: > verify_code = ''' > + #include > #include "gdbm.h" > > static datum pygdbm_fetch(GDBM_FILE gdbm_file, char *dptr, int dsize) { > @@ -86,102 +89,121 @@ > return {'dptr': ffi.new("char[]", key), 'dsize': len(key)} > > class gdbm(object): > - ll_dbm = None > + __ll_dbm = None > + > + # All public methods need to acquire the lock; all private methods > + # assume the lock is already held. Thus public methods cannot call > + # other public methods. > > def __init__(self, filename, iflags, mode): > - res = lib.gdbm_open(filename, 0, iflags, mode, ffi.NULL) > - self.size = -1 > - if not res: > - self._raise_from_errno() > - self.ll_dbm = res > + with _lock: > + res = lib.gdbm_open(filename, 0, iflags, mode, ffi.NULL) > + self.__size = -1 > + if not res: > + self.__raise_from_errno() > + self.__ll_dbm = res > > def close(self): > - if self.ll_dbm: > - lib.gdbm_close(self.ll_dbm) > - self.ll_dbm = None > + with _lock: > + if self.__ll_dbm: > + lib.gdbm_close(self.__ll_dbm) > + self.__ll_dbm = None > > - def _raise_from_errno(self): > + def __raise_from_errno(self): > if ffi.errno: > raise error(ffi.errno, os.strerror(ffi.errno)) > raise error(lib.gdbm_errno, lib.gdbm_strerror(lib.gdbm_errno)) > > def __len__(self): > - if self.size < 0: > - self.size = len(self.keys()) > - return self.size > + with _lock: > + if self.__size < 0: > + self.__size = len(self.__keys()) > + return self.__size > > def __setitem__(self, key, value): > - self._check_closed() > - self.size = -1 > - r = lib.gdbm_store(self.ll_dbm, _fromstr(key), _fromstr(value), > - lib.GDBM_REPLACE) > - if r < 0: > - self._raise_from_errno() > + with _lock: > + self.__check_closed() > + self.__size = -1 > + r = lib.gdbm_store(self.__ll_dbm, _fromstr(key), _fromstr(value), > + lib.GDBM_REPLACE) > + if r < 0: > + self.__raise_from_errno() > > def __delitem__(self, key): > - self._check_closed() > - self.size = -1 > - res = lib.gdbm_delete(self.ll_dbm, _fromstr(key)) > - if res < 0: > - raise KeyError(key) > + with _lock: > + self.__check_closed() > + self.__size = -1 > + res = lib.gdbm_delete(self.__ll_dbm, _fromstr(key)) > + if res < 0: > + raise KeyError(key) > > def __contains__(self, key): > - self._check_closed() > - key = _checkstr(key) > - return lib.pygdbm_exists(self.ll_dbm, key, len(key)) > + with _lock: > + self.__check_closed() > + key = _checkstr(key) > + return lib.pygdbm_exists(self.__ll_dbm, key, len(key)) > has_key = __contains__ > > def __getitem__(self, key): > - self._check_closed() > - key = _checkstr(key) > - drec = lib.pygdbm_fetch(self.ll_dbm, key, len(key)) > - if not drec.dptr: > - raise KeyError(key) > - res = str(ffi.buffer(drec.dptr, drec.dsize)) > - lib.free(drec.dptr) > - return res > + with _lock: > + self.__check_closed() > + key = _checkstr(key) > + drec = lib.pygdbm_fetch(self.__ll_dbm, key, len(key)) > + if not drec.dptr: > + raise KeyError(key) > + res = str(ffi.buffer(drec.dptr, drec.dsize)) > + lib.free(drec.dptr) > + return res > > - def keys(self): > - self._check_closed() > + def __keys(self): > + self.__check_closed() > l = [] > - key = lib.gdbm_firstkey(self.ll_dbm) > + key = lib.gdbm_firstkey(self.__ll_dbm) > while key.dptr: > l.append(str(ffi.buffer(key.dptr, key.dsize))) > - nextkey = lib.gdbm_nextkey(self.ll_dbm, key) > + nextkey = lib.gdbm_nextkey(self.__ll_dbm, key) > lib.free(key.dptr) > key = nextkey > return l > > + def keys(self): > + with _lock: > + return self.__keys() > + > def firstkey(self): > - self._check_closed() > - key = lib.gdbm_firstkey(self.ll_dbm) > - if key.dptr: > - res = str(ffi.buffer(key.dptr, key.dsize)) > - lib.free(key.dptr) > - return res > + with _lock: > + self.__check_closed() > + key = lib.gdbm_firstkey(self.__ll_dbm) > + if key.dptr: > + res = str(ffi.buffer(key.dptr, key.dsize)) > + lib.free(key.dptr) > + return res > > def nextkey(self, key): > - self._check_closed() > - key = lib.gdbm_nextkey(self.ll_dbm, _fromstr(key)) > - if key.dptr: > - res = str(ffi.buffer(key.dptr, key.dsize)) > - lib.free(key.dptr) > - return res > + with _lock: > + self.__check_closed() > + key = lib.gdbm_nextkey(self.__ll_dbm, _fromstr(key)) > + if key.dptr: > + res = str(ffi.buffer(key.dptr, key.dsize)) > + lib.free(key.dptr) > + return res > > def reorganize(self): > - self._check_closed() > - if lib.gdbm_reorganize(self.ll_dbm) < 0: > - self._raise_from_errno() > + with _lock: > + self.__check_closed() > + if lib.gdbm_reorganize(self.__ll_dbm) < 0: > + self.__raise_from_errno() > > - def _check_closed(self): > - if not self.ll_dbm: > + def __check_closed(self): > + if not self.__ll_dbm: > raise error(0, "GDBM object has already been closed") > > __del__ = close > > def sync(self): > - self._check_closed() > - lib.gdbm_sync(self.ll_dbm) > + with _lock: > + self.__check_closed() > + lib.gdbm_sync(self.__ll_dbm) > > def open(filename, flags='r', mode=0666): > if flags[0] == 'r': > _______________________________________________ > pypy-commit mailing list > pypy-commit at python.org > https://mail.python.org/mailman/listinfo/pypy-commit From arigo at tunes.org Thu May 7 12:20:48 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 7 May 2015 12:20:48 +0200 Subject: [pypy-dev] [pypy-commit] pypy default: The gdbm library is not thread-safe. Add a global lock. In-Reply-To: References: <20150506193916.136A61C0FE0@cobra.cs.uni-duesseldorf.de> Message-ID: Hi Fijal, On 6 May 2015 at 23:15, Maciej Fijalkowski wrote: > I was thinking, maybe instead we can add a feature to cffi "don't > release the GIL" and use that there? (it would be faster for example) It would not work here: one problem is that gdbm_*() functions set the global variable gdbm_errno, which must be read afterwards in case of error. So even if the gdbm_*() function call is done without releasing the GIL, the GIL could still be released after the function call and before we check the global variable. In other words, "don't release the GIL around this C external call because I want it to be atomic" is really a hack. It would only work if we had "with atomic:" from STM, and then it's basically the same as what I've written, if "_lock == atomic". For performance we could imagine a hack where we support "with atomic" specially in the JIT, causing both the external function call to be done without releasing the GIL, and causing all other release-the-GIL-now to be ignored. This could be implemented relatively efficiently. Unsure it's worth the double trouble---for us and for the user of locks... A bient?t, Armin. From hengha.mao at gmail.com Fri May 8 13:32:28 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Fri, 8 May 2015 19:32:28 +0800 Subject: [pypy-dev] Build pypy 2.5.1 source code on Redhat EL5.7 Message-ID: Hi, We would like to try latest pypy 2.5.1. However, we found out the downloaded binary was not compatible with our Linux enviroment (Redhat EL5.7, kernel 2.6.32). And for some reasons, we have no plans to upgrade Linux soon. I tried to build pypy 2.5.1 source code, and met some errors as below: [translation:ERROR] CompilationError: CompilationError(err=""" [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In function ?dump_section_SSL_TLSEXT_ERR_ALERT_FATAL?: [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: error: ?SSL_TLSEXT_ERR_ALERT_FATAL? undeclared (first use in this function) [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: error: (Each undeclared identifier is reported only once [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: error: for each function it appears in.) [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In function ?dump_section_SSL_TLSEXT_ERR_OK?: [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:729: error: ?SSL_TLSEXT_ERR_OK? undeclared (first use in this function) [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In function ?dump_section_TLSEXT_NAMETYPE_host_name?: [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:769: error: ?TLSEXT_NAMETYPE_host_name? undeclared (first use in this function) I checked openssl version on the system: $grep "define OPENSSL_VERSION_TEX" /usr/include/openssl/opensslv.h #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008" #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-rhel5 01 Jul 2008" OpenSSL version looks old and does not have those variables definition. Any good ideas to solve the issue? Thanks! -Ethan -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyo.arkarlwin at gmail.com Fri May 8 16:53:06 2015 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Fri, 8 May 2015 21:23:06 +0630 Subject: [pypy-dev] Build pypy 2.5.1 source code on Redhat EL5.7 In-Reply-To: References: Message-ID: can you try with portable pypy ? Hi, We would like to try latest pypy 2.5.1. However, we found out the downloaded binary was not compatible with our Linux enviroment (Redhat EL5.7, kernel 2.6.32). And for some reasons, we have no plans to upgrade Linux soon. I tried to build pypy 2.5.1 source code, and met some errors as below: [translation:ERROR] CompilationError: CompilationError(err=""" [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In function ?dump_section_SSL_TLSEXT_ERR_ALERT_FATAL?: [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: error: ?SSL_TLSEXT_ERR_ALERT_FATAL? undeclared (first use in this function) [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: error: (Each undeclared identifier is reported only once [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: error: for each function it appears in.) [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In function ?dump_section_SSL_TLSEXT_ERR_OK?: [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:729: error: ?SSL_TLSEXT_ERR_OK? undeclared (first use in this function) [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In function ?dump_section_TLSEXT_NAMETYPE_host_name?: [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:769: error: ?TLSEXT_NAMETYPE_host_name? undeclared (first use in this function) I checked openssl version on the system: $grep "define OPENSSL_VERSION_TEX" /usr/include/openssl/opensslv.h #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008" #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-rhel5 01 Jul 2008" OpenSSL version looks old and does not have those variables definition. Any good ideas to solve the issue? Thanks! -Ethan _______________________________________________ pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyo.arkarlwin at gmail.com Fri May 8 16:55:26 2015 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Fri, 8 May 2015 21:25:26 +0630 Subject: [pypy-dev] Build pypy 2.5.1 source code on Redhat EL5.7 In-Reply-To: References: Message-ID: https://github.com/squeaky-pl/portable-pypy On May 8, 2015 9:23 PM, "Phyo Arkar" wrote: > can you try with portable pypy ? > Hi, > > We would like to try latest pypy 2.5.1. > However, we found out the downloaded binary was not compatible with our > Linux enviroment (Redhat EL5.7, kernel 2.6.32). And for some reasons, we > have no plans to upgrade Linux soon. > I tried to build pypy 2.5.1 source code, and met some errors as below: > > [translation:ERROR] CompilationError: CompilationError(err=""" > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In > function ?dump_section_SSL_TLSEXT_ERR_ALERT_FATAL?: > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: > error: ?SSL_TLSEXT_ERR_ALERT_FATAL? undeclared (first use in this function) > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: > error: (Each undeclared identifier is reported only once > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: > error: for each function it appears in.) > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In > function ?dump_section_SSL_TLSEXT_ERR_OK?: > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:729: > error: ?SSL_TLSEXT_ERR_OK? undeclared (first use in this function) > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In > function ?dump_section_TLSEXT_NAMETYPE_host_name?: > [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:769: > error: ?TLSEXT_NAMETYPE_host_name? undeclared (first use in this function) > > I checked openssl version on the system: > $grep "define OPENSSL_VERSION_TEX" /usr/include/openssl/opensslv.h > #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008" > #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-rhel5 01 Jul 2008" > > OpenSSL version looks old and does not have those variables definition. > Any good ideas to solve the issue? > > Thanks! > > -Ethan > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Sat May 9 04:59:13 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Sat, 9 May 2015 10:59:13 +0800 Subject: [pypy-dev] Build pypy 2.5.1 source code on Redhat EL5.7 In-Reply-To: References: Message-ID: Great thanks! The portable pypy did work. I am very interested in how it done, but did not find detail descriptions on github page. Could you please provide the document on how to build the portable version? On Fri, May 8, 2015 at 10:55 PM, Phyo Arkar wrote: > https://github.com/squeaky-pl/portable-pypy > On May 8, 2015 9:23 PM, "Phyo Arkar" wrote: > >> can you try with portable pypy ? >> Hi, >> >> We would like to try latest pypy 2.5.1. >> However, we found out the downloaded binary was not compatible with our >> Linux enviroment (Redhat EL5.7, kernel 2.6.32). And for some reasons, we >> have no plans to upgrade Linux soon. >> I tried to build pypy 2.5.1 source code, and met some errors as below: >> >> [translation:ERROR] CompilationError: CompilationError(err=""" >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In >> function ?dump_section_SSL_TLSEXT_ERR_ALERT_FATAL?: >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: >> error: ?SSL_TLSEXT_ERR_ALERT_FATAL? undeclared (first use in this function) >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: >> error: (Each undeclared identifier is reported only once >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:719: >> error: for each function it appears in.) >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In >> function ?dump_section_SSL_TLSEXT_ERR_OK?: >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:729: >> error: ?SSL_TLSEXT_ERR_OK? undeclared (first use in this function) >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c: In >> function ?dump_section_TLSEXT_NAMETYPE_host_name?: >> [translation:ERROR] /tmp/usession-release-2.5.1-23/platcheck_31.c:769: >> error: ?TLSEXT_NAMETYPE_host_name? undeclared (first use in this function) >> >> I checked openssl version on the system: >> $grep "define OPENSSL_VERSION_TEX" /usr/include/openssl/opensslv.h >> #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-fips-rhel5 01 Jul 2008" >> #define OPENSSL_VERSION_TEXT "OpenSSL 0.9.8e-rhel5 01 Jul 2008" >> >> OpenSSL version looks old and does not have those variables definition. >> Any good ideas to solve the issue? >> >> Thanks! >> >> -Ethan >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat May 9 10:42:12 2015 From: arigo at tunes.org (Armin Rigo) Date: Sat, 9 May 2015 10:42:12 +0200 Subject: [pypy-dev] PyXxx_Malloc() in cpyext Message-ID: Hi all (mostly Amaury :-), Some C extension modules don't compile with cpyext because it is missing some of the functions like PyObject_Malloc (it has got PyMem_Malloc). I'm thinking about adding them but also simplifying the indirections and having all PyObject_Malloc(), PyMem_Malloc(), etc. just directly call malloc(), i.e. never using pypy's own obmalloc implementation. Would there be any problem with that? (We should really at some point measure if using this obmalloc implementation for the rest of PyPy makes any sense at all. My guess is no...) A bient?t, Armin. From yury at shurup.com Sat May 9 10:42:49 2015 From: yury at shurup.com (Yury V. Zaytsev) Date: Sat, 09 May 2015 10:42:49 +0200 Subject: [pypy-dev] Build pypy 2.5.1 source code on Redhat EL5.7 In-Reply-To: References: Message-ID: <1431160969.20850.31.camel@newpride> On Sat, 2015-05-09 at 10:59 +0800, Yicong Huang wrote: > > Could you please provide the document on how to build the portable > version? Have a look at the last section of the readme in the portable PyPy repository; all the magic is in the repository as well (make_portable and drive.py). -- Sincerely yours, Yury V. Zaytsev From phyo.arkarlwin at gmail.com Sat May 9 11:15:28 2015 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Sat, 9 May 2015 15:45:28 +0630 Subject: [pypy-dev] Build pypy 2.5.1 source code on Redhat EL5.7 In-Reply-To: <1431160969.20850.31.camel@newpride> References: <1431160969.20850.31.camel@newpride> Message-ID: ofcoz , it is an opensource project. On Sat, May 9, 2015 at 3:12 PM, Yury V. Zaytsev wrote: > On Sat, 2015-05-09 at 10:59 +0800, Yicong Huang wrote: > > > > Could you please provide the document on how to build the portable > > version? > > Have a look at the last section of the readme in the portable PyPy > repository; all the magic is in the repository as well (make_portable > and drive.py). > > -- > Sincerely yours, > Yury V. Zaytsev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bokr at oz.net Mon May 11 00:28:46 2015 From: bokr at oz.net (Bengt Richter) Date: Mon, 11 May 2015 00:28:46 +0200 Subject: [pypy-dev] Pycket: A Tracing JIT For a Functional Language Message-ID: Further evidence of your good rpython effects beyond pypy: http://lambda-the-ultimate.org/node/5152 http://homes.soic.indiana.edu/samth/pycket-draft.pdf in case any of you hadn't seen it yet. Kudos. From hengha.mao at gmail.com Mon May 11 10:40:52 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Mon, 11 May 2015 16:40:52 +0800 Subject: [pypy-dev] pypy-2.5.1 source code compiled errors Message-ID: Hi, I tired to compile pypy-2.5.1 source code on Redhat EL 5.7. After quite a long time, I observed final linking errors: [translation:info] Error: [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/goal/translate.py", line 318, in main [translation:info] drv.proceed(goals) [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/driver.py", line 539, in proceed [translation:info] return self._execute(goals, task_skip = self._maybe_skip()) [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/tool/taskengine.py", line 114, in _execute [translation:info] res = self._do(goal, taskcallable, *args, **kwds) [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/driver.py", line 276, in _do [translation:info] res = func() [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/driver.py", line 505, in task_compile_c [translation:info] cbuilder.compile(**kwds) [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/c/genc.py", line 375, in compile [translation:info] extra_opts) [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/platform/posix.py", line 211, in execute_makefile [translation:info] self._handle_error(returncode, stdout, stderr, path.join('make')) [translation:info] File "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/platform/__init__.py", line 151, in _handle_error [translation:info] raise CompilationError(stdout, stderr) [translation:ERROR] CompilationError: CompilationError(err=""" [translation:ERROR] data_pypy_module_cpyext_pyobject.c:101: warning: initialization from incompatible pointer type [translation:ERROR] data_pypy_module_cpyext_pyobject.c:125: warning: initialization from incompatible pointer type .. a lot of warnings about initialization from incompatible pointer type ... [translation:ERROR] /usr/bin/ld: implement.o: relocation R_X86_64_PC32 against `pypy_asm_stackwalk' can not be used when making a shared object; recompile with -fPIC [translation:ERROR] /usr/bin/ld: final link failed: Bad value [translation:ERROR] collect2: ld returned 1 exit status [translation:ERROR] make: *** [libpypy-c.so] Error 1 Shall I need to add "-fPIC"? If yes, where shall I put this flag? Thanks! -Ethan -------------- next part -------------- An HTML attachment was scrubbed... URL: From stanislav.bohm at vsb.cz Mon May 11 20:02:50 2015 From: stanislav.bohm at vsb.cz (Stanislav Bohm) Date: Mon, 11 May 2015 20:02:50 +0200 Subject: [pypy-dev] pypy + cProfile Message-ID: <5550EECA.2070804@vsb.cz> Hi I have observed that Pypy is faster when cProfile is enabled than its normal execution. I would like to ask if it is a normal behavior or a kind of anomaly? E.g. one of my benchmarks: Pypy: ~24.5s Pypy + cProfile: ~19s CPython: ~22s Tested with 2.5.1+dfsg-1~ppa1+ubuntu14.04, but I have observed a similar behavior with older versions of Pypy shipped with Ubuntu 14.10 and 15.04. My application is a pure python application that communicates through sockets. (It is a state-space analytical tool that communicates with a modified Valgrind process through TCP/IP connection). A replication of results: Build: https://github.com/spirali/aislinn (Installation instructions: http://verif.cs.vsb.cz/aislinn/doc/userguide.html#_installation) cd aislinn/tests/complex/workers ../../../bin/mpicc workers.c # build an analyzed program time pypy ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 time pypy -m cProfile ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 time python ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 Best regards, Stanislav Bohm From hengha.mao at gmail.com Tue May 12 09:06:47 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 12 May 2015 15:06:47 +0800 Subject: [pypy-dev] pypy-2.5.1 source code compiled errors In-Reply-To: References: Message-ID: The problem was solved. GCC 4.1 in Redhat EL5.7 is old. By using GCC 4.7.1, there were no such errors. On Mon, May 11, 2015 at 4:40 PM, Yicong Huang wrote: > Hi, > > I tired to compile pypy-2.5.1 source code on Redhat EL 5.7. > After quite a long time, I observed final linking errors: > > [translation:info] Error: > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/goal/translate.py", > line 318, in main > [translation:info] drv.proceed(goals) > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/driver.py", line > 539, in proceed > [translation:info] return self._execute(goals, task_skip = > self._maybe_skip()) > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/tool/taskengine.py", > line 114, in _execute > [translation:info] res = self._do(goal, taskcallable, *args, **kwds) > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/driver.py", line > 276, in _do > [translation:info] res = func() > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/driver.py", line > 505, in task_compile_c > [translation:info] cbuilder.compile(**kwds) > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/c/genc.py", line > 375, in compile > [translation:info] extra_opts) > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/platform/posix.py", > line 211, in execute_makefile > [translation:info] self._handle_error(returncode, stdout, stderr, > path.join('make')) > [translation:info] File > "/home/yicong.hyc/test/pypy-2.5.1-src/rpython/translator/platform/__init__.py", > line 151, in _handle_error > [translation:info] raise CompilationError(stdout, stderr) > [translation:ERROR] CompilationError: CompilationError(err=""" > [translation:ERROR] data_pypy_module_cpyext_pyobject.c:101: warning: > initialization from incompatible pointer type > [translation:ERROR] data_pypy_module_cpyext_pyobject.c:125: warning: > initialization from incompatible pointer type > > .. a lot of warnings about initialization from incompatible pointer type > ... > > [translation:ERROR] /usr/bin/ld: implement.o: relocation R_X86_64_PC32 > against `pypy_asm_stackwalk' can not be used when making a shared object; > recompile with -fPIC > [translation:ERROR] /usr/bin/ld: final link failed: Bad value > [translation:ERROR] collect2: ld returned 1 exit status > [translation:ERROR] make: *** [libpypy-c.so] Error 1 > > Shall I need to add "-fPIC"? > If yes, where shall I put this flag? > > Thanks! > > -Ethan > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Tue May 12 11:48:09 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 12 May 2015 17:48:09 +0800 Subject: [pypy-dev] operror-value: ('Unknown character', ('c callback', 1, 1, '\x08\n', 0)) Message-ID: Hi, I followed the document ( http://pypy.readthedocs.org/en/latest/embedding.html), to call python function from C code. The program I wrote is simple: it read python code from a local file to char*, and pass char* to pypy_execute_source(). However, I met these errors when executing the program: debug: OperationError: debug: operror-type: SyntaxError debug: operror-value: ('Unknown character', ('c callback', 1, 1, '\x08\n', 0)) Error calling pypy_execute_source! I print out char* that read from the local python file. It looks right. And the python file had no problmes to exeute with 'pypy-c'. I have no thoughts of where the error character '\x08' come from. Could anyone please help? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Wed May 13 04:42:47 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 13 May 2015 10:42:47 +0800 Subject: [pypy-dev] How to call the "same pypy function" from C for thousands of times? Message-ID: Hi, I read the document "Embedding Pypy". It introduces the method to call pypy function from C. In our scenaro, we would like to call the same pypy function from C for thousands of times or even millions of times. Following the document, the code is like this: for(i = 0; i < count; i++){ res = pypy_execute_source(pyBuffer); if(res) break; } if (res || i < count) { printf("Error calling pypy_execute_source!\n"); } However, the code is quite slow for reasons: 1. Pypy treats the same function call for a new function every time. It took time to do parsing, analysing and such a lot of work. 2. JIT could not give any benifits for the single function call. For looping 100,000 times in C code, it took about 1120.2 secondes. However, the same function executing in pypy for 100,000 times only cost about 3.7 secodes. Do we have any ideas to improve this case? For example, could we expose the function pointer in pypy to C, so that C code is able to call the same pypy function in the loop? Thanks! -Ethan -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed May 13 05:01:05 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 12 May 2015 22:01:05 -0500 Subject: [pypy-dev] How to call the "same pypy function" from C for thousands of times? In-Reply-To: References: Message-ID: <16A3A8C0-25B2-4992-BB96-F4B338E57E28@gmail.com> Couldn't you just write the loop in Python? On May 12, 2015 9:42:47 PM CDT, Yicong Huang wrote: >Hi, > >I read the document "Embedding Pypy". It introduces the method to call >pypy >function from C. >In our scenaro, we would like to call the same pypy function from C for >thousands of times or even millions of times. >Following the document, the code is like this: > > for(i = 0; i < count; i++){ > res = pypy_execute_source(pyBuffer); > if(res) > break; > } > if (res || i < count) { > printf("Error calling pypy_execute_source!\n"); > } > >However, the code is quite slow for reasons: >1. Pypy treats the same function call for a new function every time. It >took time to do parsing, analysing and such a lot of work. >2. JIT could not give any benifits for the single function call. > >For looping 100,000 times in C code, it took about 1120.2 secondes. >However, the same function executing in pypy for 100,000 times only >cost >about 3.7 secodes. > >Do we have any ideas to improve this case? >For example, could we expose the function pointer in pypy to C, so that >C >code is able to call the same pypy function in the loop? > >Thanks! > >-Ethan > > >------------------------------------------------------------------------ > >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Wed May 13 05:41:08 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 13 May 2015 11:41:08 +0800 Subject: [pypy-dev] How to call the "same pypy function" from C for thousands of times? In-Reply-To: <16A3A8C0-25B2-4992-BB96-F4B338E57E28@gmail.com> References: <16A3A8C0-25B2-4992-BB96-F4B338E57E28@gmail.com> Message-ID: The python functions usage is like UDF. The function parameters might be different from time to time. We are not able to get all parmaters we need to call at the begining. And we' ve no ideas how many times we need to call. It all depends on how user input the command at the front end. And one more question, how to get the return value from pypy function? On Wed, May 13, 2015 at 11:01 AM, Ryan Gonzalez wrote: > Couldn't you just write the loop in Python? > > On May 12, 2015 9:42:47 PM CDT, Yicong Huang wrote: > >> Hi, >> >> I read the document "Embedding Pypy". It introduces the method to call >> pypy function from C. >> In our scenaro, we would like to call the same pypy function from C for >> thousands of times or even millions of times. >> Following the document, the code is like this: >> >> for(i = 0; i < count; i++){ >> res = pypy_execute_source(pyBuffer); >> if(res) >> break; >> } >> if (res || i < count) { >> printf("Error calling pypy_execute_source!\n"); >> } >> >> However, the code is quite slow for reasons: >> 1. Pypy treats the same function call for a new function every time. It >> took time to do parsing, analysing and such a lot of work. >> 2. JIT could not give any benifits for the single function call. >> >> For looping 100,000 times in C code, it took about 1120.2 secondes. >> However, the same function executing in pypy for 100,000 times only cost >> about 3.7 secodes. >> >> Do we have any ideas to improve this case? >> For example, could we expose the function pointer in pypy to C, so that C >> code is able to call the same pypy function in the loop? >> >> Thanks! >> >> -Ethan >> >> ------------------------------ >> >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Wed May 13 07:39:50 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 13 May 2015 13:39:50 +0800 Subject: [pypy-dev] How to get the return value from pypy function for C code Message-ID: >From the document " "Embedding Pypy", we got the method to call python function from C code. But there are no examples on how to get the function return value. Does pypy have this feature? -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Wed May 13 09:25:20 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 13 May 2015 09:25:20 +0200 Subject: [pypy-dev] How to get the return value from pypy function for C code In-Reply-To: References: Message-ID: Hi, 2015-05-13 7:39 GMT+02:00 Yicong Huang : > From the document " "Embedding Pypy", we got the method to call python > function from C code. > But there are no examples on how to get the function return value. > Does pypy have this feature? > I think you need to use the function "pypy_execute_source_ptr", and pass a pointer to a C variable. >From the Python side, it's turned into an int, that you can cast back to a ffi pointer. Something like this: import cffi ffi = cffi.FFI() result_ptr = ffi.cast('double *', c_argument) result_ptr[0] = 42.0 -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Wed May 13 09:55:00 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 13 May 2015 15:55:00 +0800 Subject: [pypy-dev] How to get the return value from pypy function for C code In-Reply-To: References: Message-ID: Thanks for your advices! But your code seems to only solve the problem about how to pass the function parameter from C to Python. In the document example: char source[] = "from cffi import FFI\n\ffi = FFI()\n\@ffi.callback('int(int)')\n\def func(a):\n\ print 'Got from C %d' % a\n\ return a * 2\n\ffi.cdef('int callback(int (*func)(int));')\n\c_func = ffi.cast('int(*)(int(*)(int))', c_argument)\n\c_func(func)\n\print 'finished the Python part'\n\"; int callback(int (*func)(int)){ printf("Calling to Python, result: %d\n", func(3));} We are able to pass the function paramter '3' to python function func(a). But how could we extract the function result '6' to C code? In CPython, there is 'PyObject_CallObject', which returns python fucntion result. Does pypy have similar API? On Wed, May 13, 2015 at 3:25 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2015-05-13 7:39 GMT+02:00 Yicong Huang : > >> From the document " "Embedding Pypy", we got the method to call python >> function from C code. >> But there are no examples on how to get the function return value. >> Does pypy have this feature? >> > > > I think you need to use the function "pypy_execute_source_ptr", and pass a > pointer to a C variable. > From the Python side, it's turned into an int, that you can cast back to a > ffi pointer. > Something like this: > > import cffi > ffi = cffi.FFI() > result_ptr = ffi.cast('double *', c_argument) > result_ptr[0] = 42.0 > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Wed May 13 09:58:21 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 13 May 2015 09:58:21 +0200 Subject: [pypy-dev] How to call the "same pypy function" from C for thousands of times? In-Reply-To: References: Message-ID: Hi, 2015-05-13 4:42 GMT+02:00 Yicong Huang : > However, the code is quite slow for reasons: > 1. Pypy treats the same function call for a new function every time. It > took time to do parsing, analysing and such a lot of work. > 2. JIT could not give any benifits for the single function call. > As said in the docs, the trick is to call pypy_execute_source once, and make it return a set of C function pointers. Here is how I would do it: - in a file "interface.py" import cffi ffi = cffi.FFI() ffi.cdef(''' struct API { double (*add_numbers)(double x, double y); }; ''') # Better define callbacks at module scope, it's important to keep this object alive. @ffi.callback("double (double, double)") def add_numbers(x, y): return x + y def fill_api(ptr): api = ffi.cast("struct API*", ptr) api.add_numbers = add_numbers - From C code: struct API { double (*add_numbers)(double x, double y); }; API api; const char source[] = "import interface; interface.fill_api(c_argument)"; res = pypy_execute_source_ptr(source, &api); // Now call this in a loop: api.add_numbers(12, 13); -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Wed May 13 11:10:48 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 13 May 2015 11:10:48 +0200 Subject: [pypy-dev] How to get the return value from pypy function for C code In-Reply-To: References: Message-ID: 2015-05-13 9:55 GMT+02:00 Yicong Huang : > Thanks for your advices! > But your code seems to only solve the problem about how to pass the > function parameter from C to Python. > In the document example: > > char source[] = "from cffi import FFI\n\ffi = FFI()\n\@ffi.callback('int(int)')\n\def func(a):\n\ print 'Got from C %d' % a\n\ return a * 2\n\ffi.cdef('int callback(int (*func)(int));')\n\c_func = ffi.cast('int(*)(int(*)(int))', c_argument)\n\c_func(func)\n\print 'finished the Python part'\n\"; > int callback(int (*func)(int)){ > printf("Calling to Python, result: %d\n", func(3));} > > We are able to pass the function paramter '3' to python function func(a). > But how could we extract the function result '6' to C code? > In CPython, there is 'PyObject_CallObject', which returns python > fucntion result. > Does pypy have similar API? > But don't you have it already? the func(3) above should return the integer 6! > > On Wed, May 13, 2015 at 3:25 PM, Amaury Forgeot d'Arc > wrote: > >> Hi, >> >> 2015-05-13 7:39 GMT+02:00 Yicong Huang : >> >>> From the document " "Embedding Pypy", we got the method to call python >>> function from C code. >>> But there are no examples on how to get the function return value. >>> Does pypy have this feature? >>> >> >> >> I think you need to use the function "pypy_execute_source_ptr", and pass >> a pointer to a C variable. >> From the Python side, it's turned into an int, that you can cast back to >> a ffi pointer. >> Something like this: >> >> import cffi >> ffi = cffi.FFI() >> result_ptr = ffi.cast('double *', c_argument) >> result_ptr[0] = 42.0 >> >> -- >> Amaury Forgeot d'Arc >> > > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From omer.drow at gmail.com Wed May 13 16:24:44 2015 From: omer.drow at gmail.com (Omer Katz) Date: Wed, 13 May 2015 17:24:44 +0300 Subject: [pypy-dev] pypy + cProfile In-Reply-To: <5550EECA.2070804@vsb.cz> References: <5550EECA.2070804@vsb.cz> Message-ID: That's insane. Profiling slows down the application. Always. Are you able to reproduce this on other scripts as well? 2015-05-11 21:02 GMT+03:00 Stanislav Bohm : > Hi > > I have observed that Pypy is faster when cProfile is enabled than its > normal execution. > I would like to ask if it is a normal behavior or a kind of anomaly? > > E.g. one of my benchmarks: > > Pypy: ~24.5s > Pypy + cProfile: ~19s > CPython: ~22s > > Tested with 2.5.1+dfsg-1~ppa1+ubuntu14.04, but I have observed a similar > behavior with older versions of Pypy shipped with Ubuntu 14.10 and 15.04. > My application is a pure python application that communicates through > sockets. (It is a state-space analytical tool that communicates with a > modified Valgrind process through TCP/IP connection). > > A replication of results: > > Build: https://github.com/spirali/aislinn (Installation instructions: > http://verif.cs.vsb.cz/aislinn/doc/userguide.html#_installation) > > cd aislinn/tests/complex/workers > ../../../bin/mpicc workers.c # build an analyzed program > time pypy ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 > time pypy -m cProfile ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 > time python ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 > > Best regards, > Stanislav Bohm > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikeckennedy at gmail.com Mon May 11 19:10:19 2015 From: mikeckennedy at gmail.com (Michael Kennedy) Date: Mon, 11 May 2015 17:10:19 +0000 Subject: [pypy-dev] Be on my podcast In-Reply-To: References: Message-ID: Hi guys, does that time work? On Fri, May 1, 2015 at 6:07 PM Michael Kennedy wrote: > Great, sent you an invite. Let me know if it works! > > On Thu, Apr 30, 2015 at 9:28 AM Maciej Fijalkowski > wrote: > >> After the 25th of May >> >> On Thu, Apr 30, 2015 at 6:12 PM, Michael Kennedy >> wrote: >> > Hi Maciej, >> > >> > Accents are fine. :) Do you have some time later in May (last few >> weeks)? >> > >> > Thanks, >> > Michael >> > >> > >> > On Wed, Apr 22, 2015 at 9:33 AM Maciej Fijalkowski >> wrote: >> >> >> >> Hi Michael >> >> >> >> I'm sorry I did not reply, somehow missed it. Sure, I'm happy to be on >> >> your podcast, beware of my thick eastern european accent though :-) >> >> >> >> Cheers, >> >> fijal >> >> >> >> On Wed, Apr 15, 2015 at 8:07 PM, Michael Kennedy < >> mikeckennedy at gmail.com> >> >> wrote: >> >> > I'd love to have you guys on my podcast, Talk Python To Me. You can >> >> > learn >> >> > more here: >> >> > >> >> > http://www.talkpythontome.com/ >> >> > >> >> > Interested in being a guest? Or a couple of you even? >> >> > >> >> > Thanks! >> >> > Michael >> >> > >> >> > _______________________________________________ >> >> > pypy-dev mailing list >> >> > pypy-dev at python.org >> >> > https://mail.python.org/mailman/listinfo/pypy-dev >> >> > >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 14 05:02:02 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 14 May 2015 11:02:02 +0800 Subject: [pypy-dev] How to convert python string to C char*? Message-ID: We had a python function that return a string value. The function will callback in C code. The below is an example of the code: @ffi.callback("char *(char *, char *)") def strconcat(x, y): x1 = ffi.string(x) y1 = ffi.string(y) print x1 + y1 return x1 + y1 The error messages are: Trying to convert the result back to C: TypeError: initializer for ctype 'char *' must be a cdata pointer, not str How could we convert the return python string value to C char*? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 14 05:02:42 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 14 May 2015 11:02:42 +0800 Subject: [pypy-dev] How to call the "same pypy function" from C for thousands of times? In-Reply-To: References: Message-ID: Great thanks! The method slove our problems. On Wed, May 13, 2015 at 3:58 PM, Amaury Forgeot d'Arc wrote: > Hi, > > 2015-05-13 4:42 GMT+02:00 Yicong Huang : > >> However, the code is quite slow for reasons: >> 1. Pypy treats the same function call for a new function every time. It >> took time to do parsing, analysing and such a lot of work. >> 2. JIT could not give any benifits for the single function call. >> > > As said in the docs, the trick is to call pypy_execute_source once, and > make it return a set of C function pointers. > > Here is how I would do it: > > - in a file "interface.py" > > import cffi > > ffi = cffi.FFI() > ffi.cdef(''' > struct API { > double (*add_numbers)(double x, double y); > }; > ''') > > # Better define callbacks at module scope, it's important to keep this > object alive. > @ffi.callback("double (double, double)") > def add_numbers(x, y): > return x + y > > def fill_api(ptr): > api = ffi.cast("struct API*", ptr) > api.add_numbers = add_numbers > > > - From C code: > > struct API { > double (*add_numbers)(double x, double y); > }; > > API api; > const char source[] = "import interface; > interface.fill_api(c_argument)"; > res = pypy_execute_source_ptr(source, &api); > > // Now call this in a loop: > api.add_numbers(12, 13); > > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stanislav.bohm at vsb.cz Thu May 14 08:37:02 2015 From: stanislav.bohm at vsb.cz (Stanislav Bohm) Date: Thu, 14 May 2015 08:37:02 +0200 Subject: [pypy-dev] pypy + cProfile In-Reply-To: References: <5550EECA.2070804@vsb.cz> Message-ID: <5554428E.9070906@vsb.cz> I have tried some standard benchmarks and my other python programs, and cProfile makes things slower. Hence, Aislinn is the only program where I can reproduce the weird behavior. Stanislav Bohm On 05/13/2015 04:24 PM, Omer Katz wrote: > That's insane. Profiling slows down the application. Always. > Are you able to reproduce this on other scripts as well? > > 2015-05-11 21:02 GMT+03:00 Stanislav Bohm >: > > Hi > > I have observed that Pypy is faster when cProfile is enabled than > its normal execution. > I would like to ask if it is a normal behavior or a kind of anomaly? > > E.g. one of my benchmarks: > > Pypy: ~24.5s > Pypy + cProfile: ~19s > CPython: ~22s > > Tested with 2.5.1+dfsg-1~ppa1+ubuntu14.04, but I have observed a > similar behavior with older versions of Pypy shipped with Ubuntu > 14.10 and 15.04. My application is a pure python application that > communicates through sockets. (It is a state-space analytical tool > that communicates with a modified Valgrind process through TCP/IP > connection). > > A replication of results: > > Build: https://github.com/spirali/aislinn (Installation > instructions: > http://verif.cs.vsb.cz/aislinn/doc/userguide.html#_installation) > > cd aislinn/tests/complex/workers > ../../../bin/mpicc workers.c # build an analyzed program > time pypy ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 > time pypy -m cProfile ../../../src/aislinn/aislinn.py -p4 ./a.out > 10 90 > time python ../../../src/aislinn/aislinn.py -p4 ./a.out 10 90 > > Best regards, > Stanislav Bohm > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 14 10:13:25 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 May 2015 10:13:25 +0200 Subject: [pypy-dev] How to convert python string to C char*? In-Reply-To: References: Message-ID: Hi Yicong, (CC to the cffi mailing list) On 14 May 2015 at 05:02, Yicong Huang wrote: > We had a python function that return a string value. The function will > callback in C code. > The below is an example of the code: > > @ffi.callback("char *(char *, char *)") > def strconcat(x, y): > x1 = ffi.string(x) > y1 = ffi.string(y) > print x1 + y1 > return x1 + y1 > > The error messages are: > Trying to convert the result back to C: > TypeError: initializer for ctype 'char *' must be a cdata pointer, not str > > How could we convert the return python string value to C char*? This is not obvious, because you have problems writing it in C too: returning a "char *" is only possible when you return some constant string literal (only in C, can't do that with cffi), or when you allocate the resulting string and keep it alive for as long as the caller needs it (and you don't really know how long that is). If you have a callback to a C library that you didn't write, it must be written in its documentation. If it is not, and you really have no choice, then you have to guess. The following would work in all cases but leak the strings by never releasing them: ALL_RESULTS = [] @ffi.callback("char *(char *, char *)") def strconcat(x, y): x1 = ffi.string(x) y1 = ffi.string(y) print x1 + y1 p = ffi.new("char[]", x1 + y1) ALL_RESULTS.append(p) return p There are chances that it works as expected if you only keep alive the last returned result; try it out, but no guarantee. A bient?t, Armin. From arigo at tunes.org Thu May 14 10:21:03 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 May 2015 10:21:03 +0200 Subject: [pypy-dev] operror-value: ('Unknown character', ('c callback', 1, 1, '\x08\n', 0)) In-Reply-To: References: Message-ID: Hi Yicong, On 12 May 2015 at 11:48, Yicong Huang wrote: > The program I wrote is simple: it read python code from a local file to > char*, and pass char* to pypy_execute_source(). It works for us, so please give a complete example: ideally, a C program that can show the problem when executed (including the Python source file if it's not embedded in the C program). Maybe it is an issue with the encoding of the file not being recognized only in "pypy_execute_source"? But that's just a guess. A bient?t, Armin. From arigo at tunes.org Thu May 14 10:25:19 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 May 2015 10:25:19 +0200 Subject: [pypy-dev] How to get the return value from pypy function for C code In-Reply-To: References: Message-ID: Hi Yicong, On 13 May 2015 at 11:10, Amaury Forgeot d'Arc wrote: >> Does pypy have similar API? > > But don't you have it already? the func(3) above should return the integer 6! I would guess so too. The point of the PyPy "API" is that it is completely minimal. You have to do everything with CFFI. But then, you *can* do whatever you like with CFFI without the constrains of a particular API. A bient?t, Armin. From arigo at tunes.org Thu May 14 10:29:41 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 May 2015 10:29:41 +0200 Subject: [pypy-dev] Be on my podcast In-Reply-To: References: Message-ID: Hi Michael, On 11 May 2015 at 19:10, Michael Kennedy wrote: > Hi guys, does that time work? Just so you know, your mails are being sent to the general pypy-dev mailing list too. We can't tell when "that time" is, probably because you have used a different channel to set it up privately. Fijal is away until ~the 25th of May, so you are unlikely to get prompt answers from him. A bient?t, Armin. From hengha.mao at gmail.com Thu May 14 11:17:17 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 14 May 2015 17:17:17 +0800 Subject: [pypy-dev] operror-value: ('Unknown character', ('c callback', 1, 1, '\x08\n', 0)) In-Reply-To: References: Message-ID: Sorry, I found that's a bug of the code. In some cases, char* buffer was free by accident before passing to pypy_execute_source(). On Thu, May 14, 2015 at 4:21 PM, Armin Rigo wrote: > Hi Yicong, > > On 12 May 2015 at 11:48, Yicong Huang wrote: > > The program I wrote is simple: it read python code from a local file to > > char*, and pass char* to pypy_execute_source(). > > It works for us, so please give a complete example: ideally, a C > program that can show the problem when executed (including the Python > source file if it's not embedded in the C program). Maybe it is an > issue with the encoding of the file not being recognized only in > "pypy_execute_source"? But that's just a guess. > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 14 11:18:17 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 May 2015 11:18:17 +0200 Subject: [pypy-dev] pypy + cProfile In-Reply-To: <5554428E.9070906@vsb.cz> References: <5550EECA.2070804@vsb.cz> <5554428E.9070906@vsb.cz> Message-ID: Hi Stanislav, On 14 May 2015 at 08:37, Stanislav Bohm wrote: > I have tried some standard benchmarks and my other python programs, and > cProfile makes things slower. > Hence, Aislinn is the only program where I can reproduce the weird behavior. Then it looks likely to be a "randomness" issue: details in your program are JIT-compiled differently when cProfile is enabled. For example, cProfile makes traces longer and this can change a few heuristics. It's likely that by tweaking your program a bit randomly you can also get the same 20% variation. Not all program exhibit this kind of detail-dependent behavior, but one or two of our benchmark do: while most benchmarks have a relatively stable performance, there are a couple of them whose performance occasionally jumps between two known values. (The randomness in this case seem to be triggered by development inside PyPy rather than changes to the benchmarks, but it's the same issue.) A bient?t, Armin. From hengha.mao at gmail.com Thu May 14 11:33:23 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 14 May 2015 17:33:23 +0800 Subject: [pypy-dev] How to convert python string to C char*? In-Reply-To: References: Message-ID: Thanks, the method did work. To my understanding, "ALL_RESULTS" is used to prevent pypy GC the pointing buffer. And thus, it is the C callback function's responsibility to free the buffer. Am I right? On Thu, May 14, 2015 at 4:13 PM, Armin Rigo wrote: > Hi Yicong, > > (CC to the cffi mailing list) > > On 14 May 2015 at 05:02, Yicong Huang wrote: > > We had a python function that return a string value. The function will > > callback in C code. > > The below is an example of the code: > > > > @ffi.callback("char *(char *, char *)") > > def strconcat(x, y): > > x1 = ffi.string(x) > > y1 = ffi.string(y) > > print x1 + y1 > > return x1 + y1 > > > > The error messages are: > > Trying to convert the result back to C: > > TypeError: initializer for ctype 'char *' must be a cdata pointer, not > str > > > > How could we convert the return python string value to C char*? > > This is not obvious, because you have problems writing it in C too: > returning a "char *" is only possible when you return some constant > string literal (only in C, can't do that with cffi), or when you > allocate the resulting string and keep it alive for as long as the > caller needs it (and you don't really know how long that is). > > If you have a callback to a C library that you didn't write, it must > be written in its documentation. If it is not, and you really have no > choice, then you have to guess. The following would work in all cases > but leak the strings by never releasing them: > > ALL_RESULTS = [] > @ffi.callback("char *(char *, char *)") > def strconcat(x, y): > x1 = ffi.string(x) > y1 = ffi.string(y) > print x1 + y1 > p = ffi.new("char[]", x1 + y1) > ALL_RESULTS.append(p) > return p > > There are chances that it works as expected if you only keep alive the > last returned result; try it out, but no guarantee. > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 14 11:40:04 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 14 May 2015 02:40:04 -0700 (PDT) Subject: [pypy-dev] How to convert python string to C char*? In-Reply-To: References: Message-ID: <11208b55-6b52-4cf3-8026-430ead3b4cbc@googlegroups.com> Thanks, the method did work. To my understanding, "ALL_RESULTS" is used to prevent pypy GC the pointing buffer. And thus, it is the C callback function's responsibility to free the buffer. Am I right? On Thursday, May 14, 2015 at 4:14:07 PM UTC+8, Armin Rigo wrote: > > Hi Yicong, > > (CC to the cffi mailing list) > > On 14 May 2015 at 05:02, Yicong Huang > > wrote: > > We had a python function that return a string value. The function will > > callback in C code. > > The below is an example of the code: > > > > @ffi.callback("char *(char *, char *)") > > def strconcat(x, y): > > x1 = ffi.string(x) > > y1 = ffi.string(y) > > print x1 + y1 > > return x1 + y1 > > > > The error messages are: > > Trying to convert the result back to C: > > TypeError: initializer for ctype 'char *' must be a cdata pointer, not > str > > > > How could we convert the return python string value to C char*? > > This is not obvious, because you have problems writing it in C too: > returning a "char *" is only possible when you return some constant > string literal (only in C, can't do that with cffi), or when you > allocate the resulting string and keep it alive for as long as the > caller needs it (and you don't really know how long that is). > > If you have a callback to a C library that you didn't write, it must > be written in its documentation. If it is not, and you really have no > choice, then you have to guess. The following would work in all cases > but leak the strings by never releasing them: > > ALL_RESULTS = [] > @ffi.callback("char *(char *, char *)") > def strconcat(x, y): > x1 = ffi.string(x) > y1 = ffi.string(y) > print x1 + y1 > p = ffi.new("char[]", x1 + y1) > ALL_RESULTS.append(p) > return p > > There are chances that it works as expected if you only keep alive the > last returned result; try it out, but no guarantee. > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 14 12:29:19 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 14 May 2015 12:29:19 +0200 Subject: [pypy-dev] How to convert python string to C char*? In-Reply-To: References: Message-ID: Hi Yicong, On 14 May 2015 at 11:33, Yicong Huang wrote: > To my understanding, "ALL_RESULTS" is used to prevent pypy GC the pointing > buffer. > And thus, it is the C callback function's responsibility to free the buffer. > Am I right? No. In my example I just keep storing ffi.new("char[]") objects in the list, so this list grows forever and leaks. You really need to see the documentation of the C code for which you are making a callback. Maybe it says that the callback needs to return a string obtained by ``malloc()``, and it will be ``free()``-ed for you. In that case you need to call ``lib.malloc()`` from the library obtained by cffi, after adding "char *malloc(size_t);" to the cdef(). But I can't guess if I'm correct; it depends on the C code. A bient?t, Armin. From santagada at gmail.com Thu May 14 21:51:48 2015 From: santagada at gmail.com (Leonardo Santagada) Date: Thu, 14 May 2015 16:51:48 -0300 Subject: [pypy-dev] pypy + cProfile In-Reply-To: References: <5550EECA.2070804@vsb.cz> <5554428E.9070906@vsb.cz> Message-ID: I'm thinking that maybe the JIT is trying to compile some trace and then throw it away, by using cprofile it is not even trying... that's why it is actually slower than cpython to begin with. Can this be the case Armin? It is a variation of what you said, but a more problematic one I think. Maybe there is some dinamism or code generation in your program that is tricking the JIT Stanislav Bohm. On Thu, May 14, 2015 at 6:18 AM, Armin Rigo wrote: > Hi Stanislav, > > On 14 May 2015 at 08:37, Stanislav Bohm wrote: > > I have tried some standard benchmarks and my other python programs, and > > cProfile makes things slower. > > Hence, Aislinn is the only program where I can reproduce the weird > behavior. > > Then it looks likely to be a "randomness" issue: details in your > program are JIT-compiled differently when cProfile is enabled. For > example, cProfile makes traces longer and this can change a few > heuristics. It's likely that by tweaking your program a bit randomly > you can also get the same 20% variation. > > Not all program exhibit this kind of detail-dependent behavior, but > one or two of our benchmark do: while most benchmarks have a > relatively stable performance, there are a couple of them whose > performance occasionally jumps between two known values. (The > randomness in this case seem to be triggered by development inside > PyPy rather than changes to the benchmarks, but it's the same issue.) > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Leonardo Santagada -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri May 15 08:05:37 2015 From: arigo at tunes.org (Armin Rigo) Date: Fri, 15 May 2015 08:05:37 +0200 Subject: [pypy-dev] pypy + cProfile In-Reply-To: References: <5550EECA.2070804@vsb.cz> <5554428E.9070906@vsb.cz> Message-ID: Hi Leonardo, On 14 May 2015 at 21:51, Leonardo Santagada wrote: > I'm thinking that maybe the JIT is trying to compile some trace and then > throw it away, by using cprofile it is not even trying... that's why it is > actually slower than cpython to begin with. Can this be the case Armin? That's idle speculation at this point. I can't say more without trying to look exactly at what occurs in this example, which I'm unsure is a good way to spend my time --- but I'd be glad to if Stanislav says he really wants to know in more details. A bient?t, Armin. From stanislav.bohm at vsb.cz Fri May 15 09:44:38 2015 From: stanislav.bohm at vsb.cz (Stanislav Bohm) Date: Fri, 15 May 2015 09:44:38 +0200 Subject: [pypy-dev] pypy + cProfile In-Reply-To: References: <5550EECA.2070804@vsb.cz> <5554428E.9070906@vsb.cz> Message-ID: <5555A3E6.2030908@vsb.cz> On 05/14/2015 11:18 AM, Armin Rigo wrote: > Hi Stanislav, > > Then it looks likely to be a "randomness" issue: details in your > program are JIT-compiled differently when cProfile is enabled. For > example, cProfile makes traces longer and this can change a few > heuristics. It's likely that by tweaking your program a bit randomly > you can also get the same 20% variation. > > Thank you for the answer. I have tried to benchmark some older versions of my program before series of program core modifications, but results have still the same characteristics, pypy is slower than CPython, pypy+cProfile is faster than CPython. Hence I am afraid that a random tweaking does not help me. On 05/15/2015 08:05 AM, Armin Rigo wrote: > That's idle speculation at this point. I can't say more without > trying to look exactly at what occurs in this example, which I'm > unsure is a good way to spend my time --- but I'd be glad to if > Stanislav says he really wants to know in more details. I am very curious to discover the root of the problem and I am ready for a cooperation. At least, I would like to end with an argument as "--jit MAGIC_VALUE=123" than with enabling cProfile for a better performance:) But I completely understand that you do not have time to dig into my program. Best regards, Stanislav Bohm From hengha.mao at gmail.com Fri May 15 14:07:58 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Fri, 15 May 2015 20:07:58 +0800 Subject: [pypy-dev] Could PyPy be embeded running in sanbox? Message-ID: In the document, we see PyPy could be embeded in C++ code as the following: pypy_execute_source_ptr(source, &api); The source is a string of python code. But I think by default the code does not run in sandbox. Are there any methods to run the code in sandbox? -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri May 15 16:56:28 2015 From: arigo at tunes.org (Armin Rigo) Date: Fri, 15 May 2015 16:56:28 +0200 Subject: [pypy-dev] Could PyPy be embeded running in sanbox? In-Reply-To: References: Message-ID: Hi Yicong, No, PyPy cannot be embedded running sandbox. The way we present embedding is by using the cffi module on the Python code; but this module is not available at all in a sandboxed PyPy (as it allows random invalid things to occur). If you really want to use the sandbox, you need to consider a completely different approach: run ``pypy-sandbox`` as a subprocess, as documented on the sandboxing section on http://pypy.org/ . But note that sandboxing is a prototype that was never really finished (even though it should be safe). What would be missing for you would be the whole C code that talks over stdin/stdout to the subprocess. Alternatively, we could imagine to embed a ``pypy-sandbox.so`` as a dynamically linked library. There is no API for that, though. It would need to be defined and cannot be just an ``pypy_execute_source()`` function, because the Python side cannot use cffi. A bient?t, Armin. From hengha.mao at gmail.com Sat May 16 07:42:06 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Sat, 16 May 2015 13:42:06 +0800 Subject: [pypy-dev] Could PyPy be embeded running in sanbox? In-Reply-To: References: Message-ID: Hi Armin, Thanks for the answer! I quite agree with you that cfii is very powerful and should not include in sandbox. However, could we consider include a small subset of cffi? Considering the typical user usage, I think the below features are sufficient to satisfiy most of cases: 1. Convert common objects from C to Python (function paramerts) 2. Expose python function pointer and allow C to callback 3. Return common objects to C (function return value) Here common object means long, double, int and etc. , pointers and struct alike complex objects are forbidden. And Python code is not allowed to import external lib or call any external function in imbeded sanbox. In this way, do you think sanbox might still work properly? And "pypy-sandbox.so" looks very attractive. I haven't figured out the strong difference of calling "pypy_execute_source()" from running with "pypy-sandbox". They both accept a string of python code. One difference I thought of is the string from "pypy_execute_source()" is C char* string, and need cffi to convert to python string. Are there any othe concerns to provide a "pypy_execute_source()" API for "pypy-sandbox.so"? On Fri, May 15, 2015 at 10:56 PM, Armin Rigo wrote: > Hi Yicong, > > No, PyPy cannot be embedded running sandbox. The way we present > embedding is by using the cffi module on the Python code; but this > module is not available at all in a sandboxed PyPy (as it allows > random invalid things to occur). > > If you really want to use the sandbox, you need to consider a > completely different approach: run ``pypy-sandbox`` as a subprocess, > as documented on the sandboxing section on http://pypy.org/ . But > note that sandboxing is a prototype that was never really finished > (even though it should be safe). What would be missing for you would > be the whole C code that talks over stdin/stdout to the subprocess. > > Alternatively, we could imagine to embed a ``pypy-sandbox.so`` as a > dynamically linked library. There is no API for that, though. It > would need to be defined and cannot be just an > ``pypy_execute_source()`` function, because the Python side cannot use > cffi. > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Mon May 18 05:54:11 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Mon, 18 May 2015 11:54:11 +0800 Subject: [pypy-dev] Fail to build pypy-c-sandbox Message-ID: I followed the document to build pypy-c-sandbox based on souce code 2.5.1: rpython/bin/rpython -O2 --sandbox pypy/goal/targetpypystandalone.py In the stdout, I did observe sandbox option is true: [translation] [translation] [translation] check_str_without_nul = True [translation] gc = generation [translation] gcrootfinder = shadowstack [translation] gctransformer = framework [translation] list_comprehension_operations = True [translation] rweakref = True [translation] sandbox = True [translation] shared = True [translation] thread = False [translation] withsmallfuncsets = 5 However, after the build process completed, only pypy-c generated and I did not find pypy-c-sandbox. [platform:execute] make -j 5 in /tmp/usession-master-16/testing_1 [platform:WARNING] pypy_module__warnings_interp_warnings.c: In function ?pypy_g_normalize_module?: [platform:WARNING] pypy_module__warnings_interp_warnings.c:8565:5: warning: assuming signed overflow does not occur when assuming that (X - c) > X is always false [-Wstrict-overflow] [translation:info] copied: /home/yicong.hyc/odps-pypy/libpypy-c.so [translation:info] usession directory: /tmp/usession-master-16 [translation:info] created: /home/yicong.hyc/odps-pypy/pypy-c [1ebe2] translation-task} [Timer] Timings: [Timer] annotate --- 309.6 s [Timer] rtype_lltype --- 355.1 s [Timer] backendopt_lltype --- 179.8 s [Timer] stackcheckinsertion_lltype --- 17.5 s [Timer] database_c --- 208.2 s [Timer] source_c --- 179.9 s [Timer] compile_c --- 102.6 s [Timer] =========================================== [Timer] Total: --- 1352.7 s Grep "sandbox" in the log, there were some warnings, but no errors: [sandbox:WARNING] Not Implemented: sandboxing for external function 'RPython_StartupCode' [sandbox:WARNING] Not Implemented: sandboxing for external function 'pypy_debug_catch_fatal_exception' [sandbox:WARNING] Not Implemented: SomeImpossibleValue() [sandbox:WARNING] Not Implemented: SomeImpossibleValue() [sandbox:WARNING] Not Implemented: sandboxing for external function 'clock_gettime' [sandbox:WARNING] Not Implemented: sandboxing for external function 'clock_getres' [sandbox:WARNING] Not Implemented: sandboxing for external function 'fcntl' -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostia.lopuhin at gmail.com Mon May 18 11:07:35 2015 From: kostia.lopuhin at gmail.com (=?UTF-8?B?0JrQvtGB0YLRjyDQm9C+0L/Rg9GF0LjQvQ==?=) Date: Mon, 18 May 2015 12:07:35 +0300 Subject: [pypy-dev] Fail to build pypy-c-sandbox In-Reply-To: References: Message-ID: At least in pypy 2.3.1 pypy-c *is* the sandbox, there is no pypy-c-sandbox produced. 2015-05-18 6:54 GMT+03:00 Yicong Huang : > I followed the document to build pypy-c-sandbox based on souce code 2.5.1: > > rpython/bin/rpython -O2 --sandbox pypy/goal/targetpypystandalone.py > > In the stdout, I did observe sandbox option is true: > > [translation] [translation] > [translation] check_str_without_nul = True > [translation] gc = generation > [translation] gcrootfinder = shadowstack > [translation] gctransformer = framework > [translation] list_comprehension_operations = True > [translation] rweakref = True > [translation] sandbox = True > [translation] shared = True > [translation] thread = False > [translation] withsmallfuncsets = 5 > > However, after the build process completed, only pypy-c generated and I did > not find pypy-c-sandbox. > > [platform:execute] make -j 5 in /tmp/usession-master-16/testing_1 > [platform:WARNING] pypy_module__warnings_interp_warnings.c: In function > ?pypy_g_normalize_module?: > [platform:WARNING] pypy_module__warnings_interp_warnings.c:8565:5: warning: > assuming signed overflow does not occur when assuming that (X - c) > X is > always false [-Wstrict-overflow] > [translation:info] copied: /home/yicong.hyc/odps-pypy/libpypy-c.so > [translation:info] usession directory: /tmp/usession-master-16 > [translation:info] created: /home/yicong.hyc/odps-pypy/pypy-c > [1ebe2] translation-task} > [Timer] Timings: > [Timer] annotate --- 309.6 s > [Timer] rtype_lltype --- 355.1 s > [Timer] backendopt_lltype --- 179.8 s > [Timer] stackcheckinsertion_lltype --- 17.5 s > [Timer] database_c --- 208.2 s > [Timer] source_c --- 179.9 s > [Timer] compile_c --- 102.6 s > [Timer] =========================================== > [Timer] Total: --- 1352.7 s > > Grep "sandbox" in the log, there were some warnings, but no errors: > > [sandbox:WARNING] Not Implemented: sandboxing for external function > 'RPython_StartupCode' > [sandbox:WARNING] Not Implemented: sandboxing for external function > 'pypy_debug_catch_fatal_exception' > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() > [sandbox:WARNING] Not Implemented: sandboxing for external function > 'clock_gettime' > [sandbox:WARNING] Not Implemented: sandboxing for external function > 'clock_getres' > [sandbox:WARNING] Not Implemented: sandboxing for external function 'fcntl' > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From hengha.mao at gmail.com Mon May 18 12:19:54 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Mon, 18 May 2015 18:19:54 +0800 Subject: [pypy-dev] Fail to build pypy-c-sandbox In-Reply-To: References: Message-ID: Thanks for the answer! As you mentioned, the generated "pypy-c" was acutally "pypy-c-sandbox". But the binary had some problems to run: $./pypy-c sll_os.ll_os_getenv(sPYPY_GENERATIONGC_NURSERY Fatal error during initialization: out of memory Aborted Alternatively, we tried to run with pypy_interact.py: pypy/sandbox/pypy_interact.py --tmp=/home/yicong.hyc/test/ ./pypy-c /home/yicong.hyc/test/twoTagCombination.py 'import site' failed Traceback (most recent call last): File "/app_main.py", line 75, in run_toplevel IOError: [Errno 2] No such file or directory: '/home/yicong.hyc/test/twoTagCombination.py' [Subprocess exit code: 1] Why the subprocess could not find the python script? We did check that the python script exist. On Mon, May 18, 2015 at 5:07 PM, ????? ??????? wrote: > At least in pypy 2.3.1 pypy-c *is* the sandbox, there is no > pypy-c-sandbox produced. > > 2015-05-18 6:54 GMT+03:00 Yicong Huang : > > I followed the document to build pypy-c-sandbox based on souce code > 2.5.1: > > > > rpython/bin/rpython -O2 --sandbox pypy/goal/targetpypystandalone.py > > > > In the stdout, I did observe sandbox option is true: > > > > [translation] [translation] > > [translation] check_str_without_nul = True > > [translation] gc = generation > > [translation] gcrootfinder = shadowstack > > [translation] gctransformer = framework > > [translation] list_comprehension_operations = True > > [translation] rweakref = True > > [translation] sandbox = True > > [translation] shared = True > > [translation] thread = False > > [translation] withsmallfuncsets = 5 > > > > However, after the build process completed, only pypy-c generated and I > did > > not find pypy-c-sandbox. > > > > [platform:execute] make -j 5 in /tmp/usession-master-16/testing_1 > > [platform:WARNING] pypy_module__warnings_interp_warnings.c: In function > > ?pypy_g_normalize_module?: > > [platform:WARNING] pypy_module__warnings_interp_warnings.c:8565:5: > warning: > > assuming signed overflow does not occur when assuming that (X - c) > X is > > always false [-Wstrict-overflow] > > [translation:info] copied: /home/yicong.hyc/odps-pypy/libpypy-c.so > > [translation:info] usession directory: /tmp/usession-master-16 > > [translation:info] created: /home/yicong.hyc/odps-pypy/pypy-c > > [1ebe2] translation-task} > > [Timer] Timings: > > [Timer] annotate --- 309.6 s > > [Timer] rtype_lltype --- 355.1 s > > [Timer] backendopt_lltype --- 179.8 s > > [Timer] stackcheckinsertion_lltype --- 17.5 s > > [Timer] database_c --- 208.2 s > > [Timer] source_c --- 179.9 s > > [Timer] compile_c --- 102.6 s > > [Timer] =========================================== > > [Timer] Total: --- 1352.7 s > > > > Grep "sandbox" in the log, there were some warnings, but no errors: > > > > [sandbox:WARNING] Not Implemented: sandboxing for external function > > 'RPython_StartupCode' > > [sandbox:WARNING] Not Implemented: sandboxing for external function > > 'pypy_debug_catch_fatal_exception' > > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() > > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() > > [sandbox:WARNING] Not Implemented: sandboxing for external function > > 'clock_gettime' > > [sandbox:WARNING] Not Implemented: sandboxing for external function > > 'clock_getres' > > [sandbox:WARNING] Not Implemented: sandboxing for external function > 'fcntl' > > > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostia.lopuhin at gmail.com Mon May 18 12:43:10 2015 From: kostia.lopuhin at gmail.com (=?UTF-8?B?0JrQvtGB0YLRjyDQm9C+0L/Rg9GF0LjQvQ==?=) Date: Mon, 18 May 2015 13:43:10 +0300 Subject: [pypy-dev] Fail to build pypy-c-sandbox In-Reply-To: References: Message-ID: You can check sandbox like this: $ ../sandbox/pypy_interact.py pypy-c >>>> import os >>>> os.getcwd() '/tmp' As for the second failed command I think the correct usage will be (from the example here http://pypy.readthedocs.org/en/latest/sandbox.html#howto): pypy/sandbox/pypy_interact.py --tmp=/home/yicong.hyc/test/ ./pypy-c /tmp/twoTagCombination.py 2015-05-18 13:19 GMT+03:00 Yicong Huang : > Thanks for the answer! > As you mentioned, the generated "pypy-c" was acutally "pypy-c-sandbox". > But the binary had some problems to run: > > $./pypy-c > s ll_os.ll_os_getenv( s PYPY_GENERATIONGC_NURSERY > Fatal error during initialization: out of memory > Aborted > > Alternatively, we tried to run with pypy_interact.py: > pypy/sandbox/pypy_interact.py --tmp=/home/yicong.hyc/test/ ./pypy-c > /home/yicong.hyc/test/twoTagCombination.py > > 'import site' failed > Traceback (most recent call last): > File "/app_main.py", line 75, in run_toplevel > IOError: [Errno 2] No such file or directory: > '/home/yicong.hyc/test/twoTagCombination.py' > [Subprocess exit code: 1] > > Why the subprocess could not find the python script? We did check that the > python script exist. > > On Mon, May 18, 2015 at 5:07 PM, ????? ??????? > wrote: >> >> At least in pypy 2.3.1 pypy-c *is* the sandbox, there is no >> pypy-c-sandbox produced. >> >> 2015-05-18 6:54 GMT+03:00 Yicong Huang : >> > I followed the document to build pypy-c-sandbox based on souce code >> > 2.5.1: >> > >> > rpython/bin/rpython -O2 --sandbox pypy/goal/targetpypystandalone.py >> > >> > In the stdout, I did observe sandbox option is true: >> > >> > [translation] [translation] >> > [translation] check_str_without_nul = True >> > [translation] gc = generation >> > [translation] gcrootfinder = shadowstack >> > [translation] gctransformer = framework >> > [translation] list_comprehension_operations = True >> > [translation] rweakref = True >> > [translation] sandbox = True >> > [translation] shared = True >> > [translation] thread = False >> > [translation] withsmallfuncsets = 5 >> > >> > However, after the build process completed, only pypy-c generated and I >> > did >> > not find pypy-c-sandbox. >> > >> > [platform:execute] make -j 5 in /tmp/usession-master-16/testing_1 >> > [platform:WARNING] pypy_module__warnings_interp_warnings.c: In function >> > ?pypy_g_normalize_module?: >> > [platform:WARNING] pypy_module__warnings_interp_warnings.c:8565:5: >> > warning: >> > assuming signed overflow does not occur when assuming that (X - c) > X >> > is >> > always false [-Wstrict-overflow] >> > [translation:info] copied: /home/yicong.hyc/odps-pypy/libpypy-c.so >> > [translation:info] usession directory: /tmp/usession-master-16 >> > [translation:info] created: /home/yicong.hyc/odps-pypy/pypy-c >> > [1ebe2] translation-task} >> > [Timer] Timings: >> > [Timer] annotate --- 309.6 s >> > [Timer] rtype_lltype --- 355.1 s >> > [Timer] backendopt_lltype --- 179.8 s >> > [Timer] stackcheckinsertion_lltype --- 17.5 s >> > [Timer] database_c --- 208.2 s >> > [Timer] source_c --- 179.9 s >> > [Timer] compile_c --- 102.6 s >> > [Timer] =========================================== >> > [Timer] Total: --- 1352.7 s >> > >> > Grep "sandbox" in the log, there were some warnings, but no errors: >> > >> > [sandbox:WARNING] Not Implemented: sandboxing for external function >> > 'RPython_StartupCode' >> > [sandbox:WARNING] Not Implemented: sandboxing for external function >> > 'pypy_debug_catch_fatal_exception' >> > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() >> > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() >> > [sandbox:WARNING] Not Implemented: sandboxing for external function >> > 'clock_gettime' >> > [sandbox:WARNING] Not Implemented: sandboxing for external function >> > 'clock_getres' >> > [sandbox:WARNING] Not Implemented: sandboxing for external function >> > 'fcntl' >> > >> > >> > >> > _______________________________________________ >> > pypy-dev mailing list >> > pypy-dev at python.org >> > https://mail.python.org/mailman/listinfo/pypy-dev >> > > > From hengha.mao at gmail.com Mon May 18 15:09:35 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Mon, 18 May 2015 21:09:35 +0800 Subject: [pypy-dev] Fail to build pypy-c-sandbox In-Reply-To: References: Message-ID: In the document, it said all the script's dependency should copy along to the directory. Do all python lib scripts need to copy as well? We tried to run one script with sandbox and observed the below errors: 'import site' failed Traceback (most recent call last): File "/app_main.py", line 75, in run_toplevel File "/tmp/twoTagCombination.py", line 2, in import hashlib File "/bin/lib-python/2.7/hashlib.py", line 149, in import logging File "/bin/lib-python/2.7/logging/__init__.py", line 26, in import sys, os, time, cStringIO, traceback, warnings, weakref, collections ImportError: No module named time [Subprocess exit code: 1] We didn't put hashlib.py to the direcoty, but it looks like sandbox is able to find hashlib.py. However, it miss the dependency of "time" module. How shall we solve the issue? On Mon, May 18, 2015 at 6:43 PM, ????? ??????? wrote: > You can check sandbox like this: > $ ../sandbox/pypy_interact.py pypy-c > >>>> import os > >>>> os.getcwd() > '/tmp' > > As for the second failed command I think the correct usage will be > (from the example here > http://pypy.readthedocs.org/en/latest/sandbox.html#howto): > pypy/sandbox/pypy_interact.py --tmp=/home/yicong.hyc/test/ ./pypy-c > /tmp/twoTagCombination.py > > 2015-05-18 13:19 GMT+03:00 Yicong Huang : > > Thanks for the answer! > > As you mentioned, the generated "pypy-c" was acutally "pypy-c-sandbox". > > But the binary had some problems to run: > > > > $./pypy-c > > s ll_os.ll_os_getenv( s PYPY_GENERATIONGC_NURSERY > > Fatal error during initialization: out of memory > > Aborted > > > > Alternatively, we tried to run with pypy_interact.py: > > pypy/sandbox/pypy_interact.py --tmp=/home/yicong.hyc/test/ ./pypy-c > > /home/yicong.hyc/test/twoTagCombination.py > > > > 'import site' failed > > Traceback (most recent call last): > > File "/app_main.py", line 75, in run_toplevel > > IOError: [Errno 2] No such file or directory: > > '/home/yicong.hyc/test/twoTagCombination.py' > > [Subprocess exit code: 1] > > > > Why the subprocess could not find the python script? We did check that > the > > python script exist. > > > > On Mon, May 18, 2015 at 5:07 PM, ????? ??????? > > > wrote: > >> > >> At least in pypy 2.3.1 pypy-c *is* the sandbox, there is no > >> pypy-c-sandbox produced. > >> > >> 2015-05-18 6:54 GMT+03:00 Yicong Huang : > >> > I followed the document to build pypy-c-sandbox based on souce code > >> > 2.5.1: > >> > > >> > rpython/bin/rpython -O2 --sandbox pypy/goal/targetpypystandalone.py > >> > > >> > In the stdout, I did observe sandbox option is true: > >> > > >> > [translation] [translation] > >> > [translation] check_str_without_nul = True > >> > [translation] gc = generation > >> > [translation] gcrootfinder = shadowstack > >> > [translation] gctransformer = framework > >> > [translation] list_comprehension_operations = True > >> > [translation] rweakref = True > >> > [translation] sandbox = True > >> > [translation] shared = True > >> > [translation] thread = False > >> > [translation] withsmallfuncsets = 5 > >> > > >> > However, after the build process completed, only pypy-c generated and > I > >> > did > >> > not find pypy-c-sandbox. > >> > > >> > [platform:execute] make -j 5 in /tmp/usession-master-16/testing_1 > >> > [platform:WARNING] pypy_module__warnings_interp_warnings.c: In > function > >> > ?pypy_g_normalize_module?: > >> > [platform:WARNING] pypy_module__warnings_interp_warnings.c:8565:5: > >> > warning: > >> > assuming signed overflow does not occur when assuming that (X - c) > X > >> > is > >> > always false [-Wstrict-overflow] > >> > [translation:info] copied: /home/yicong.hyc/odps-pypy/libpypy-c.so > >> > [translation:info] usession directory: /tmp/usession-master-16 > >> > [translation:info] created: /home/yicong.hyc/odps-pypy/pypy-c > >> > [1ebe2] translation-task} > >> > [Timer] Timings: > >> > [Timer] annotate --- 309.6 s > >> > [Timer] rtype_lltype --- 355.1 s > >> > [Timer] backendopt_lltype --- 179.8 s > >> > [Timer] stackcheckinsertion_lltype --- 17.5 s > >> > [Timer] database_c --- 208.2 s > >> > [Timer] source_c --- 179.9 s > >> > [Timer] compile_c --- 102.6 s > >> > [Timer] =========================================== > >> > [Timer] Total: --- 1352.7 s > >> > > >> > Grep "sandbox" in the log, there were some warnings, but no errors: > >> > > >> > [sandbox:WARNING] Not Implemented: sandboxing for external function > >> > 'RPython_StartupCode' > >> > [sandbox:WARNING] Not Implemented: sandboxing for external function > >> > 'pypy_debug_catch_fatal_exception' > >> > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() > >> > [sandbox:WARNING] Not Implemented: SomeImpossibleValue() > >> > [sandbox:WARNING] Not Implemented: sandboxing for external function > >> > 'clock_gettime' > >> > [sandbox:WARNING] Not Implemented: sandboxing for external function > >> > 'clock_getres' > >> > [sandbox:WARNING] Not Implemented: sandboxing for external function > >> > 'fcntl' > >> > > >> > > >> > > >> > _______________________________________________ > >> > pypy-dev mailing list > >> > pypy-dev at python.org > >> > https://mail.python.org/mailman/listinfo/pypy-dev > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From kostia.lopuhin at gmail.com Mon May 18 15:18:55 2015 From: kostia.lopuhin at gmail.com (=?UTF-8?B?0JrQvtGB0YLRjyDQm9C+0L/Rg9GF0LjQvQ==?=) Date: Mon, 18 May 2015 16:18:55 +0300 Subject: [pypy-dev] Fail to build pypy-c-sandbox In-Reply-To: References: Message-ID: 2015-05-18 16:09 GMT+03:00 Yicong Huang : > In the document, it said all the script's dependency should copy along to > the directory. > Do all python lib scripts need to copy as well? No, available stdlib modules should work. > We didn't put hashlib.py to the direcoty, but it looks like sandbox is able > to find hashlib.py. > However, it miss the dependency of "time" module. > How shall we solve the issue? > Not all stdlib modules are included with sandbox by default (and not all will work), and I don't think a general solution exists. For example, we needed datetime.datetime.str[fp]time functions, and what we do (for 2.3.1) is translate with "--withmod-time --withmod-struct" options added, and then in order to use datatime.datetime module we bundle a pure-python replacement. You might also need to add some support for functions that do external calls. From hengha.mao at gmail.com Tue May 19 07:29:46 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 19 May 2015 13:29:46 +0800 Subject: [pypy-dev] Exclude modules and functions from building libpypy.so Message-ID: For pypy 2.5.1, the default buiding of libpypy-c.so is about 99MB. To my understanding, libypypy.so includes standard python libs. If we don't need some of those libs, could we exclude libs from building? For one thing, we might be able to get smaller libpypy.so lib. And for the other, we might exclude some *dangerous* lib that we don't need. ( Certainly, the more safer way is to use sandbox). >From pypy/config/pypyoption.py file, we found the below code: working_modules.update([ "_socket", "unicodedata", "mmap", "fcntl", "_locale", "pwd", "time" , "select", "zipimport", "_lsprof", "crypt", "signal", "_rawffi", "termios", "zlib", "bz2", "struct", "_hashlib", "_md5", "_sha", "_minimal_curses", "cStringIO", "thread", "itertools", "pyexpat", "_ssl", "cpyext", "array", "binascii", "_multiprocessing", '_warnings', "_collections", "_multibytecodec", "micronumpy", "_continuation", "_cffi_backend", "_csv", "cppyy", "_pypyjson" ]) If we would like to exclude some modules, could we modify this list? Further, is it possible to exclude some functions from modules? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue May 19 11:11:27 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 19 May 2015 11:11:27 +0200 Subject: [pypy-dev] Exclude modules and functions from building libpypy.so In-Reply-To: References: Message-ID: there are compilation options like --withoutmod-_md5 and so on. No, you can't disable single functions. On Tue, May 19, 2015 at 7:29 AM, Yicong Huang wrote: > For pypy 2.5.1, the default buiding of libpypy-c.so is about 99MB. > To my understanding, libypypy.so includes standard python libs. > If we don't need some of those libs, could we exclude libs from building? > For one thing, we might be able to get smaller libpypy.so lib. > And for the other, we might exclude some *dangerous* lib that we don't need. > ( Certainly, the more safer way is to use sandbox). > > From pypy/config/pypyoption.py file, we found the below code: > working_modules.update([ > "_socket", "unicodedata", "mmap", "fcntl", "_locale", "pwd", "time" , > "select", "zipimport", "_lsprof", "crypt", "signal", "_rawffi", > "termios", > "zlib", "bz2", "struct", "_hashlib", "_md5", "_sha", "_minimal_curses", > "cStringIO", "thread", "itertools", "pyexpat", "_ssl", "cpyext", > "array", > "binascii", "_multiprocessing", '_warnings', "_collections", > "_multibytecodec", "micronumpy", "_continuation", "_cffi_backend", > "_csv", "cppyy", "_pypyjson" > ]) > > If we would like to exclude some modules, could we modify this list? > Further, is it possible to exclude some functions from modules? > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From Manuel.Serrano at inria.fr Tue May 19 11:34:55 2015 From: Manuel.Serrano at inria.fr (Manuel.Serrano at inria.fr) Date: Tue, 19 May 2015 11:34:55 +0200 Subject: [pypy-dev] [CFP] DLS 2015 Message-ID: <2015519113455.555b03bf@redrock.inria.fr> ----------------------------- C A L L F O R P A P E R S ----------------------------- ======== DLS 2015 =========== 11th Dynamic Languages Symposium 2015 October, 2015 Pittsburgh, Pennsylvania, United States http://DLS2015.inria.fr Co-located with SPLASH 2015 In association with ACM SIGPLAN The 11th Dynamic Languages Symposium (DLS) at SPLASH 2015 is the premier forum for researchers and practitioners to share knowledge and research on dynamic languages, their implementation, and applications. The influence of dynamic languages -- from Lisp to Smalltalk to Python to Javascript -- on real-world practice and research continues to grow. DLS 2015 invites high quality papers reporting original research, innovative contributions, or experience related to dynamic languages, their implementation, and applications. Accepted papers will be published in the ACM Digital Library, and freely available for 2 weeks before and after the event itself. Areas of interest include but are not limited to: Innovative language features and implementation techniques Development and platform support, tools Interesting applications Domain-oriented programming Very late binding, dynamic composition, and run-time adaptation Reflection and meta-programming Software evolution Language symbiosis and multi-paradigm languages Dynamic optimization Hardware support Experience reports and case studies Educational approaches and perspectives Semantics of dynamic languages == Invited Speaker == DLS is pleased to announce a talk by the following invited speaker: Eelco Visser: Declare your Language. == Submissions and proceedings == Submissions should not have been published previously nor under review at other events. Research papers should describe work that advances the current state of the art. Experience papers should be of broad interest and should describe insights gained from substantive practical applications. The program committee will evaluate each contributed paper based on its relevance, significance, clarity, length, and originality. Papers are to be submitted electronically at http://www.easychair.org/conferences?conf=dls15 in PDF format. Submissions must be in the ACM format (see http://www.sigplan.org/authorInformation.htm) and not exceed 12 pages. Authors are reminded that brevity is a virtue. DLS 2015 will run a two-phase reviewing process to help authors make their final papers the best that they can be. After the first round of reviews, papers will be rejected, conditionally accepted, or unconditionally accepted. Conditionally accepted papers will be given a list of issues raised by reviewers. Authors will then submit a revised version of the paper with a cover letter explaining how they have or why they have not addressed these issues. The reviewers will then consider the cover letter and revised paper and recommend final acceptance or rejection. Accepted papers will be published in the ACM Digital Library. Important dates Abstract Submissions: Sun 7 Jun 2015 Full Submissions: Sun 15 Jun 2015 First phase notification: Mon 27 Jul Revisions due: Mon 3 Aug Final notification: Mon 17 Aug Camera ready: Fri 21 21 Aug Program chair Manuel Serrano, Inria Sophia-Antipolis, dls15 at easychair.org Program committee Carl Friedrich Bolz, DE William R. Cook, UTexas, USA Jonathan Edwards, MIT, USA John Field, Google, USA Matt Flatt, USA Elisa Gonzalez Boix, Vrije Universiteit, BE Robert Hirschfeld, Hasso-Plattner-Institut Potsdam, DE Benjamin Livshits, Microsoft, USA Crista Lopes, UC Irvine, USA Kevin Millikin, Google, DN James Noble, Victoria University of Wellington, NZ Manuel Serrano, Inria, FR (General chair) Didier Verna, EPITA, FR Jan Vitek, Purdue, USA Joe Politz, Brown University, USA Olivier Tardieu, IBM, USA From arigo at tunes.org Tue May 19 15:18:27 2015 From: arigo at tunes.org (Armin Rigo) Date: Tue, 19 May 2015 15:18:27 +0200 Subject: [pypy-dev] Could PyPy be embeded running in sanbox? In-Reply-To: References: Message-ID: Hi Yicong, On 16 May 2015 at 07:42, Yicong Huang wrote: > I quite agree with you that cfii is very powerful and should not include in > sandbox. > However, could we consider include a small subset of cffi? No. Anything that cffi enables lets you read random memory, or write random memory, or call random places, where "random" is defined as "can be freely controlled by the Python code". For example, if you want a have a Python callback that accepts a "char *" string, you can then then a object and use it to read and write arbitrary memory anywhere (by reading and writing chars at a large offset from this pointer object). A bient?t, Armin. From hengha.mao at gmail.com Tue May 19 17:06:51 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 19 May 2015 23:06:51 +0800 Subject: [pypy-dev] For embedding pypy, how to deal with None value Message-ID: The document Embedding PyPy shows good examples, but did not tell how to deal with None value. As Python support None value, it might bring some troubles: 1. How to pass NULL from C to python for None? 2. If python function return None, how to handle the issue in C? For the above cases, we observed errors like these: >From cffi callback : Trying to convert the result back to C: TypeError: expected integer, got NoneType object -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Tue May 19 18:27:28 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 20 May 2015 00:27:28 +0800 Subject: [pypy-dev] For embedding pypy, how to deal with None value In-Reply-To: References: Message-ID: One approach we thought of is to use additional int pointer to indicate whether the value is None. ffi.cdef('''struct API { double (*add_numbers)(double x, double y); int *ret}; And then wrap the python function, if the return value is None, set ret[0] to 1. On Tue, May 19, 2015 at 11:06 PM, Yicong Huang wrote: > The document Embedding PyPy shows good examples, but did not tell how to > deal with None value. > As Python support None value, it might bring some troubles: > 1. How to pass NULL from C to python for None? > 2. If python function return None, how to handle the issue in C? > > For the above cases, we observed errors like these: > > From cffi callback : > Trying to convert the result back to C: > TypeError: expected integer, got NoneType object > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Tue May 19 18:35:43 2015 From: yury at shurup.com (Yury V. Zaytsev) Date: Tue, 19 May 2015 18:35:43 +0200 Subject: [pypy-dev] Exclude modules and functions from building libpypy.so In-Reply-To: References: Message-ID: <1432053343.23427.689.camel@newpride> On Tue, 2015-05-19 at 13:29 +0800, Yicong Huang wrote: > For pypy 2.5.1, the default buiding of libpypy-c.so is about 99MB. Did you check whether it includes the debug information or not? If yes (which looks quite likely, judging by the size), then you can strip it and this alone will get you a huge gain at practically no cost. -- Sincerely yours, Yury V. Zaytsev From rymg19 at gmail.com Tue May 19 20:09:29 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 19 May 2015 13:09:29 -0500 Subject: [pypy-dev] For embedding pypy, how to deal with None value In-Reply-To: References: Message-ID: <822BA4E9-4FD7-4EAA-B478-3E5FB67EE25D@gmail.com> Why not return a pointer to a double? Like (UNTESTED!): d = ffi.new('double*') d[0] = 9.0 return d Then you could return None, which is converted to a C NULL (same thing other way around). On May 19, 2015 11:27:28 AM CDT, Yicong Huang wrote: >One approach we thought of is to use additional int pointer to indicate >whether the value is None. > >ffi.cdef('''struct API { double (*add_numbers)(double x, double y); > > int *ret}; > >And then wrap the python function, if the return value is None, set >ret[0] to 1. > > > >On Tue, May 19, 2015 at 11:06 PM, Yicong Huang >wrote: > >> The document Embedding PyPy shows good examples, but did not tell how >to >> deal with None value. >> As Python support None value, it might bring some troubles: >> 1. How to pass NULL from C to python for None? >> 2. If python function return None, how to handle the issue in C? >> >> For the above cases, we observed errors like these: >> >> From cffi callback : >> Trying to convert the result back to C: >> TypeError: expected integer, got NoneType object >> >> >> > > >------------------------------------------------------------------------ > >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed May 20 08:39:57 2015 From: arigo at tunes.org (Armin Rigo) Date: Wed, 20 May 2015 08:39:57 +0200 Subject: [pypy-dev] For embedding pypy, how to deal with None value In-Reply-To: <822BA4E9-4FD7-4EAA-B478-3E5FB67EE25D@gmail.com> References: <822BA4E9-4FD7-4EAA-B478-3E5FB67EE25D@gmail.com> Message-ID: Hi Ryan, On 19 May 2015 at 20:09, Ryan Gonzalez wrote: > Why not return a pointer to a double? Like (UNTESTED!): > > d = ffi.new('double*') > d[0] = 9.0 > return d This doesn't work! You can't return a ffi.new() pointer, because the 'd' variable is not kept alive. >>> The document Embedding PyPy shows good examples, but did not tell how to >>> deal with None value. I'm still unsure I understand the question. You need to know at least a bit of C in order to use cffi. In C you can't write a function that returns either a double or NULL. First start by thinking about a valid C interface, and then use cffi to implement it. Maybe you want something like int foo(int argument, double *result); which returns a status code as an integer (e.g. 0=ok, -1=error), and in the ok case, fills in "*result". A bient?t, Armin. From hengha.mao at gmail.com Wed May 20 09:54:49 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 20 May 2015 15:54:49 +0800 Subject: [pypy-dev] For embedding pypy, how to deal with None value In-Reply-To: References: <822BA4E9-4FD7-4EAA-B478-3E5FB67EE25D@gmail.com> Message-ID: Yes, C could *not* have a function that returns either a double or NULL. But python could. In addition, python could have a function paramter that is either int or None. The problem is we might get a python function, and we would like to call this python function in C. We're not sure whether this function return int or return None. Follow the document ,we might define the function callback as below: @ffi.callback("int (int, int)") However, sometimes the python function will return None, which cause Operation Error from CFFI. On the other hand, it is valid to python for receving the function parameter as None. How could we pass NULL to python function via ffi.callback definition? On Wed, May 20, 2015 at 2:39 PM, Armin Rigo wrote: > Hi Ryan, > > On 19 May 2015 at 20:09, Ryan Gonzalez wrote: > > Why not return a pointer to a double? Like (UNTESTED!): > > > > d = ffi.new('double*') > > d[0] = 9.0 > > return d > > This doesn't work! You can't return a ffi.new() pointer, because the > 'd' variable is not kept alive. > > >>> The document Embedding PyPy shows good examples, but did not tell how > to > >>> deal with None value. > > I'm still unsure I understand the question. You need to know at least > a bit of C in order to use cffi. In C you can't write a function that > returns either a double or NULL. First start by thinking about a > valid C interface, and then use cffi to implement it. Maybe you want > something like > > int foo(int argument, double *result); > > which returns a status code as an integer (e.g. 0=ok, -1=error), and > in the ok case, fills in "*result". > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Wed May 20 10:46:02 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 20 May 2015 10:46:02 +0200 Subject: [pypy-dev] For embedding pypy, how to deal with None value In-Reply-To: References: <822BA4E9-4FD7-4EAA-B478-3E5FB67EE25D@gmail.com> Message-ID: 2015-05-20 9:54 GMT+02:00 Yicong Huang : > Yes, C could *not* have a function that returns either a double or NULL. > But python could. > In addition, python could have a function paramter that is either int or > None. > > The problem is we might get a python function, and we would like to call > this python function in C. > We're not sure whether this function return int or return None. > Follow the document ,we might define the function callback as below: > @ffi.callback("int (int, int)") > However, sometimes the python function will return None, which cause > Operation Error from CFFI. > > On the other hand, it is valid to python for receving the function > parameter as None. > How could we pass NULL to python function via ffi.callback definition? > cffi basic usage is to call C functions from Python, and is was designed for this. C is a simple language. Your use case is the opposite: call Python from C, and it's going to be more complex because Python types are way more flexible than C. To call from C a Python function that can return multiple types, you will have to design (in C) some way to embed these types in a single value, and more code to manipulate this value. For example, the CPython API uses the PyObject* type to represent anything, and has more than 400 functions like PyFloat_AsDouble. Yes, this is involved: cffi.new_handle() can be used to return any Python object to C, but the handle needs to stay alive on the Python side, so a pool of returned handles is necessary, but it needs to be cleared sometimes, and there will be many helper functions... I once wrote some code to demonstrate this, I need to clean it a bit and put it in the docs. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Wed May 20 11:44:23 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 20 May 2015 17:44:23 +0800 Subject: [pypy-dev] Cannot instantiate ctype 'struct *' of unknown size Message-ID: Hi, The below is the code we met troubles; ffi.cdef(''' struct API { struct api_ret* (*pyudf)(char* str); }; ''') ffi.cdef(''' struct api_ret { char *ret1; char *ret2; char *ret3; }; ''') result = ffi.new('struct api_ret*') At first, we defined struct API with a function pointer that return a pointer to struct api_ret. And then we defined api_ret, and tried to new the object. The problem seems to happen when the function pointer and new structure are together. The below code had no problems. ffi.cdef(''' struct API { struct api_ret* (*pyudf)(char* str); }; ''') ffi.cdef(''' struct api_ret2 { char *ret1; char *ret2; char *ret3; }; ''') #we used different name api_ret2 to work around the issue. result = ffi.new('struct api_ret2*') -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Wed May 20 11:51:27 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 20 May 2015 11:51:27 +0200 Subject: [pypy-dev] Cannot instantiate ctype 'struct *' of unknown size In-Reply-To: References: Message-ID: 2015-05-20 11:44 GMT+02:00 Yicong Huang : > Hi, > > The below is the code we met troubles; > > ffi.cdef(''' > struct API { > struct api_ret* (*pyudf)(char* str); > }; > ''') > > ffi.cdef(''' > struct api_ret { > char *ret1; > char *ret2; > char *ret3; > }; > ''') > > result = ffi.new('struct api_ret*') > This code works for me. What's the issue exactly? > > At first, we defined struct API with a function pointer that return a > pointer to struct api_ret. > And then we defined api_ret, and tried to new the object. > The problem seems to happen when the function pointer and new structure > are together. > The below code had no problems. > > ffi.cdef(''' > struct API { > struct api_ret* (*pyudf)(char* str); > }; > ''') > > ffi.cdef(''' > struct api_ret2 { > char *ret1; > char *ret2; > char *ret3; > }; > ''') > #we used different name api_ret2 to work around the issue. > result = ffi.new('struct api_ret2*') > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Wed May 20 12:03:14 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Wed, 20 May 2015 18:03:14 +0800 Subject: [pypy-dev] Cannot instantiate ctype 'struct *' of unknown size In-Reply-To: References: Message-ID: Oh, great thanks for you help! Finally, I found out the real issue. The whold code is this: @ffi.callback("struct api_ret*(char *)") def udf_api(arg0): return None ffi.cdef(''' struct API { struct api_ret* (*pyudf)(char* str); }; ''') ffi.cdef(''' struct api_ret { char *ret1; char *ret2; char *ret3; }; ''') result = ffi.new('struct api_ret*') And we should put the function callback annotion below struct api_ret definition. :) On Wed, May 20, 2015 at 5:51 PM, Amaury Forgeot d'Arc wrote: > 2015-05-20 11:44 GMT+02:00 Yicong Huang : > >> Hi, >> >> The below is the code we met troubles; >> >> ffi.cdef(''' >> struct API { >> struct api_ret* (*pyudf)(char* str); >> }; >> ''') >> >> ffi.cdef(''' >> struct api_ret { >> char *ret1; >> char *ret2; >> char *ret3; >> }; >> ''') >> >> result = ffi.new('struct api_ret*') >> > > This code works for me. What's the issue exactly? > > >> >> At first, we defined struct API with a function pointer that return a >> pointer to struct api_ret. >> And then we defined api_ret, and tried to new the object. >> The problem seems to happen when the function pointer and new structure >> are together. >> The below code had no problems. >> >> ffi.cdef(''' >> struct API { >> struct api_ret* (*pyudf)(char* str); >> }; >> ''') >> >> ffi.cdef(''' >> struct api_ret2 { >> char *ret1; >> char *ret2; >> char *ret3; >> }; >> ''') >> #we used different name api_ret2 to work around the issue. >> result = ffi.new('struct api_ret2*') >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From amauryfa at gmail.com Wed May 20 12:27:25 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 20 May 2015 12:27:25 +0200 Subject: [pypy-dev] Cannot instantiate ctype 'struct *' of unknown size In-Reply-To: References: Message-ID: Haha, but there is a bug in cffi I think. I just filed https://bitbucket.org/cffi/cffi/issue/193/forward-declaration-of-struct-does-not with a simpler test case. 2015-05-20 12:03 GMT+02:00 Yicong Huang : > Oh, great thanks for you help! > Finally, I found out the real issue. > The whold code is this: > > @ffi.callback("struct api_ret*(char *)") > def udf_api(arg0): > return None > > ffi.cdef(''' > struct API { > struct api_ret* (*pyudf)(char* str); > }; > ''') > > ffi.cdef(''' > struct api_ret { > char *ret1; > char *ret2; > char *ret3; > }; > ''') > > result = ffi.new('struct api_ret*') > > And we should put the function callback annotion below struct api_ret > definition. :) > > On Wed, May 20, 2015 at 5:51 PM, Amaury Forgeot d'Arc > wrote: > >> 2015-05-20 11:44 GMT+02:00 Yicong Huang : >> >>> Hi, >>> >>> The below is the code we met troubles; >>> >>> ffi.cdef(''' >>> struct API { >>> struct api_ret* (*pyudf)(char* str); >>> }; >>> ''') >>> >>> ffi.cdef(''' >>> struct api_ret { >>> char *ret1; >>> char *ret2; >>> char *ret3; >>> }; >>> ''') >>> >>> result = ffi.new('struct api_ret*') >>> >> >> This code works for me. What's the issue exactly? >> >> >>> >>> At first, we defined struct API with a function pointer that return a >>> pointer to struct api_ret. >>> And then we defined api_ret, and tried to new the object. >>> The problem seems to happen when the function pointer and new structure >>> are together. >>> The below code had no problems. >>> >>> ffi.cdef(''' >>> struct API { >>> struct api_ret* (*pyudf)(char* str); >>> }; >>> ''') >>> >>> ffi.cdef(''' >>> struct api_ret2 { >>> char *ret1; >>> char *ret2; >>> char *ret3; >>> }; >>> ''') >>> #we used different name api_ret2 to work around the issue. >>> result = ffi.new('struct api_ret2*') >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> https://mail.python.org/mailman/listinfo/pypy-dev >>> >>> >> >> >> -- >> Amaury Forgeot d'Arc >> > > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Wed May 20 14:35:32 2015 From: arigo at tunes.org (Armin Rigo) Date: Wed, 20 May 2015 14:35:32 +0200 Subject: [pypy-dev] Cannot instantiate ctype 'struct *' of unknown size In-Reply-To: References: Message-ID: Hi Yicong, hi Amaury, On 20 May 2015 at 12:27, Amaury Forgeot d'Arc wrote: > Haha, but there is a bug in cffi I think. Thanks for the bug report. Fixed in 2dfaf4b4f0aa. Armin From rymg19 at gmail.com Wed May 20 18:30:34 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 20 May 2015 11:30:34 -0500 Subject: [pypy-dev] For embedding pypy, how to deal with None value In-Reply-To: References: <822BA4E9-4FD7-4EAA-B478-3E5FB67EE25D@gmail.com> Message-ID: <7AF1259C-B307-4978-888A-CFE8F5BF0251@gmail.com> On May 20, 2015 1:39:57 AM CDT, Armin Rigo wrote: >Hi Ryan, > >On 19 May 2015 at 20:09, Ryan Gonzalez wrote: >> Why not return a pointer to a double? Like (UNTESTED!): >> >> d = ffi.new('double*') >> d[0] = 9.0 >> return d > >This doesn't work! You can't return a ffi.new() pointer, because the >'d' variable is not kept alive. *slaps self* Right. I feel smart... > >>>> The document Embedding PyPy shows good examples, but did not tell >how to >>>> deal with None value. > >I'm still unsure I understand the question. You need to know at least >a bit of C in order to use cffi. In C you can't write a function that >returns either a double or NULL. First start by thinking about a >valid C interface, and then use cffi to implement it. Maybe you want >something like > > int foo(int argument, double *result); > >which returns a status code as an integer (e.g. 0=ok, -1=error), and >in the ok case, fills in "*result". > > >A bient?t, > >Armin. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From hengha.mao at gmail.com Thu May 21 05:12:38 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 21 May 2015 11:12:38 +0800 Subject: [pypy-dev] Could PyPy callback C code with C function pointer? Message-ID: With PyPy "pypy_execute_source_ptr()", we could pass a C function pointer to python code. The funtion pointer was initilized in C code. Could PyPy use this function pointer callback to C code? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 21 05:27:34 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 21 May 2015 11:27:34 +0800 Subject: [pypy-dev] Could PyPy callback C code with C function pointer? In-Reply-To: References: Message-ID: Great. It did work. :) On Thu, May 21, 2015 at 11:12 AM, Yicong Huang wrote: > With PyPy "pypy_execute_source_ptr()", we could pass a C function pointer > to python code. > The funtion pointer was initilized in C code. > Could PyPy use this function pointer callback to C code? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From antti.makinen at simosol.fi Thu May 21 10:40:09 2015 From: antti.makinen at simosol.fi (=?utf-8?Q?Antti_M=C3=A4kinen?=) Date: Thu, 21 May 2015 11:40:09 +0300 Subject: [pypy-dev] datetime.date support in NumPyPy Message-ID: <3DD8DD1E-57FF-4F25-A5DC-2A08D0CB10F4@simosol.fi> Is there any estimate on when NumPyPy will support datetime.date objects in arrays? Regards, Antti M?kinen -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 21 11:15:52 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 21 May 2015 17:15:52 +0800 Subject: [pypy-dev] Cannot instantiate ctype 'struct *' of unknown size In-Reply-To: References: Message-ID: Great thanks for quick fix! :) On Wed, May 20, 2015 at 8:35 PM, Armin Rigo wrote: > Hi Yicong, hi Amaury, > > On 20 May 2015 at 12:27, Amaury Forgeot d'Arc wrote: > > Haha, but there is a bug in cffi I think. > > Thanks for the bug report. Fixed in 2dfaf4b4f0aa. > > Armin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronan.lamy at gmail.com Fri May 22 01:22:22 2015 From: ronan.lamy at gmail.com (Ronan Lamy) Date: Fri, 22 May 2015 00:22:22 +0100 Subject: [pypy-dev] datetime.date support in NumPyPy In-Reply-To: <3DD8DD1E-57FF-4F25-A5DC-2A08D0CB10F4@simosol.fi> References: <3DD8DD1E-57FF-4F25-A5DC-2A08D0CB10F4@simosol.fi> Message-ID: <555E68AE.4000102@gmail.com> Le 21/05/15 09:40, Antti M?kinen a ?crit : > Is there any estimate on when NumPyPy will support datetime.date objects > in arrays? Well, the object dtype is already supported in the default branch, so you can put datetime.date objects in arrays. If you were thinking of the np.datetime64 and np.timedelta64 dtypes, they aren't supported yet, and we're not planning on adding them in the immediate future unless there's a compelling use case. From hengha.mao at gmail.com Fri May 22 05:55:08 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Fri, 22 May 2015 11:55:08 +0800 Subject: [pypy-dev] ImportError, No module named pycparser Message-ID: Hi, I built pypy with some modules cut off, but met the errors as below for executing python scripts: debug: OperationError: debug: operror-type: ImportError debug: operror-value: No module named pycparser Which modules do pycparser depends on? I found the module might in /lib_pypy/cffi/_pycparser. But I did not exclude cffi module, and it did not have any problems of importing cffi. The modules I cut of are: ssl bz2 crypt _csv _minimal_curse fcntl _io mmap parser pyexpat select _socket _ssl zlib pwd And the list of used modules for the build: [usemodules] __pypy__ = True _ast = True _cffi_backend = True _codecs = True _collections = True _continuation = True _hashlib = True _locale = True _lsprof = True _md5 = True _minimal_curses = True _multibytecodec = True _multiprocessing = True _pickle_support = True _pypyjson = True _random = True _rawffi = True _sha = True _sre = True _testing = True _weakref = True array = True binascii = True cStringIO = True cmath = True cppyy = True cpyext = True errno = True gc = True imp = True marshal = True math = True micronumpy = True operator = True signal = True struct = True symbol = True termios = True thread = True time = True token = True unicodedata = True zipimport = True -------------- next part -------------- An HTML attachment was scrubbed... URL: From antti.makinen at simosol.fi Fri May 22 09:07:18 2015 From: antti.makinen at simosol.fi (=?utf-8?Q?Antti_M=C3=A4kinen?=) Date: Fri, 22 May 2015 10:07:18 +0300 Subject: [pypy-dev] datetime.date support in NumPyPy In-Reply-To: <555E68AE.4000102@gmail.com> References: <3DD8DD1E-57FF-4F25-A5DC-2A08D0CB10F4@simosol.fi> <555E68AE.4000102@gmail.com> Message-ID: <47CE23B6-D9C3-4198-A5C4-D7D218D89D0A@simosol.fi> Thanks for the answer! I don't know if I'm missing something, but initializing arrays with dtype=object seems to fail. I'm using PyPy 2.5.1 on OS X 10.10 and Numpy from the PyPy's Numpy fork. I'm assuming the "master [MAIN BRANCH]" is the same as the "default branch". Initializing the arrays fails like this: >>>> import numpy >>>> numpy.zeros(2, dtype=object) Traceback (most recent call last): File "", line 1, in NotImplementedError: cannot create dtype with type 'object' Numpy was installed like this: git clone https://bitbucket.org/pypy/numpy.git; cd numpy; pypy setup.py install > On 22 May 2015, at 02:22, Ronan Lamy wrote: > > Le 21/05/15 09:40, Antti M?kinen a ?crit : >> Is there any estimate on when NumPyPy will support datetime.date objects >> in arrays? > > Well, the object dtype is already supported in the default branch, so you can put datetime.date objects in arrays. If you were thinking of the np.datetime64 and np.timedelta64 dtypes, they aren't supported yet, and we're not planning on adding them in the immediate future unless there's a compelling use case. > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Fri May 22 09:10:53 2015 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 22 May 2015 10:10:53 +0300 Subject: [pypy-dev] Windows: pypyw.exe? In-Reply-To: References: <54D26D2D.6010603@gmail.com> Message-ID: <555ED67D.8060505@gmail.com> An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Fri May 22 09:29:07 2015 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 22 May 2015 10:29:07 +0300 Subject: [pypy-dev] Windows: pypyw.exe? In-Reply-To: <555ED67D.8060505@gmail.com> References: <54D26D2D.6010603@gmail.com> <555ED67D.8060505@gmail.com> Message-ID: <555EDAC3.1080901@gmail.com> On 22/05/15 10:10, Matti Picus wrote: > > > On 06/02/15 13:14, Amaury Forgeot d'Arc wrote: > CPython3 has this code to set sys.stdout to None in this case: > https://hg.python.org/cpython/file/v3.3.4/Python/pythonrun.c#l1083 > > But that's probably only for python3, python2 chose to not change > anything: > http://bugs.python.org/issue706263 > I merged pypyw which simply creates another exe, in line with what > Armin suggested. Amaury, it is not clear to me how stdout is set to > None since AFAICT is_valid_fd(stdout) passes, at least in the rpython > test I tried. To reproduce: run test_shared in > translator/c/test_standalone.py, then run the resulting test-1w.exe as > > test-1.exe a b 2>err.txt > > It will print a rpython exception to err.txt indicating that it failed > in the actual write, not in the is_valid_fd(stdout) test just before > the write. > > Matti Ahh, is_valid_fd in cpython also checks dummy_fd = dup(fd), which we do not do (in rposix.py). Sorry for the noise. Matti From hengha.mao at gmail.com Fri May 22 09:49:05 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Fri, 22 May 2015 15:49:05 +0800 Subject: [pypy-dev] ImportError, No module named pycparser In-Reply-To: References: Message-ID: We found out it needs module 'select' to load '_pycparser' module: >>>> from cffi import _pycparser Traceback (most recent call last): File "", line 1, in File "/home/yicong.hyc/pypy/lib_pypy/cffi/_pycparser/__init__.py", line 13, in from subprocess import Popen, PIPE File "/home/yicong.hyc/pypy/lib-python/2.7/subprocess.py", line 427, in import select ImportError: No module named select Could we build _pycparser into libpypy.so, so that we don't need 'select' to load the module? On Fri, May 22, 2015 at 11:55 AM, Yicong Huang wrote: > Hi, > > I built pypy with some modules cut off, but met the errors as below for > executing python scripts: > > debug: OperationError: > debug: operror-type: ImportError > debug: operror-value: No module named pycparser > > Which modules do pycparser depends on? > I found the module might in /lib_pypy/cffi/_pycparser. But I did not > exclude cffi module, and it did not have any problems of importing cffi. > > The modules I cut of are: > ssl > bz2 > crypt > _csv > _minimal_curse > fcntl > _io > mmap > parser > pyexpat > select > _socket > _ssl > zlib > pwd > > And the list of used modules for the build: > [usemodules] > __pypy__ = True > _ast = True > _cffi_backend = True > _codecs = True > _collections = True > _continuation = True > _hashlib = True > _locale = True > _lsprof = True > _md5 = True > _minimal_curses = True > _multibytecodec = True > _multiprocessing = True > _pickle_support = True > _pypyjson = True > _random = True > _rawffi = True > _sha = True > _sre = True > _testing = True > _weakref = True > array = True > binascii = True > cStringIO = True > cmath = True > cppyy = True > cpyext = True > errno = True > gc = True > imp = True > marshal = True > math = True > micronumpy = True > operator = True > signal = True > struct = True > symbol = True > termios = True > thread = True > time = True > token = True > unicodedata = True > zipimport = True > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Fri May 22 09:51:44 2015 From: arigo at tunes.org (Armin Rigo) Date: Fri, 22 May 2015 09:51:44 +0200 Subject: [pypy-dev] ImportError, No module named pycparser In-Reply-To: References: Message-ID: Hi Yicong, On 22 May 2015 at 09:49, Yicong Huang wrote: > from subprocess import Popen, PIPE > File "/home/yicong.hyc/pypy/lib-python/2.7/subprocess.py", line 427, in > > import select > ImportError: No module named select Then anything using "subprocess" requires the select module, and most programs out there will need the "subprocess" module. If you really want to remove this small innocent module, you are on your own. A bient?t, Armin. From matti.picus at gmail.com Fri May 22 12:26:42 2015 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 22 May 2015 13:26:42 +0300 Subject: [pypy-dev] datetime.date support in NumPyPy In-Reply-To: <47CE23B6-D9C3-4198-A5C4-D7D218D89D0A@simosol.fi> References: <3DD8DD1E-57FF-4F25-A5DC-2A08D0CB10F4@simosol.fi> <555E68AE.4000102@gmail.com> <47CE23B6-D9C3-4198-A5C4-D7D218D89D0A@simosol.fi> Message-ID: <555F0462.6010902@gmail.com> On 22/05/15 10:07, Antti M?kinen wrote: > Thanks for the answer! I don't know if I'm missing something, but > initializing arrays with dtype=object seems to fail. I'm using PyPy > 2.5.1 on OS X 10.10 and Numpy from the PyPy's Numpy fork. I'm assuming > the "master [MAIN BRANCH]" is the same as the "default branch". > > pypy-dev mailing list pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev You will need a pypy from after the merge of the object-dtype2 branch (post-release-2.5.1), unfortunately we have no working macosx buildbots at the moment so you would have to translate it for yourself. 2.6.0 should be out in two weeks or so with the object dtype. Help with finding a functioning macosx buildbot would be appreciated Matti From antti.m.makinen at gmail.com Fri May 22 12:35:22 2015 From: antti.m.makinen at gmail.com (=?windows-1252?Q?Antti_M=E4kinen?=) Date: Fri, 22 May 2015 13:35:22 +0300 Subject: [pypy-dev] datetime.date support in NumPyPy In-Reply-To: <555F0462.6010902@gmail.com> References: <3DD8DD1E-57FF-4F25-A5DC-2A08D0CB10F4@simosol.fi> <555E68AE.4000102@gmail.com> <47CE23B6-D9C3-4198-A5C4-D7D218D89D0A@simosol.fi> <555F0462.6010902@gmail.com> Message-ID: Great news! I'll wait for the 2.6.0 release then :) > On 22 May 2015, at 13:26, Matti Picus wrote: > > > > On 22/05/15 10:07, Antti M?kinen wrote: >> Thanks for the answer! I don't know if I'm missing something, but initializing arrays with dtype=object seems to fail. I'm using PyPy 2.5.1 on OS X 10.10 and Numpy from the PyPy's Numpy fork. I'm assuming the "master [MAIN BRANCH]" is the same as the "default branch". >> >> pypy-dev mailing list pypy-dev at python.org https://mail.python.org/mailman/listinfo/pypy-dev > You will need a pypy from after the merge of the object-dtype2 branch (post-release-2.5.1), unfortunately we have no working macosx buildbots at the moment so you would have to translate it for yourself. > > 2.6.0 should be out in two weeks or so with the object dtype. > > Help with finding a functioning macosx buildbot would be appreciated > Matti > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From matti.picus at gmail.com Fri May 22 15:18:19 2015 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 22 May 2015 16:18:19 +0300 Subject: [pypy-dev] PyPy OSX buildbot needed Message-ID: <555F2C9B.80405@gmail.com> this is an explicit call for help. We do not seem to be able to keep a nightly OSX buildbot functioning, even one that builds once a week or so and can be made available for major merges/releases would be better than none. If you have a lead to a modern OSX machine that can donate cycles to PyPy, please respond or join on IRC at #pypy Thanks From kostia.lopuhin at gmail.com Fri May 22 15:26:29 2015 From: kostia.lopuhin at gmail.com (=?UTF-8?B?0JrQvtGB0YLRjyDQm9C+0L/Rg9GF0LjQvQ==?=) Date: Fri, 22 May 2015 16:26:29 +0300 Subject: [pypy-dev] PyPy OSX buildbot needed In-Reply-To: <555F2C9B.80405@gmail.com> References: <555F2C9B.80405@gmail.com> Message-ID: Hello Matti! I can do weekly builds and tests on weekends or on request (most of the time). 2015-05-22 16:18 GMT+03:00 Matti Picus : > this is an explicit call for help. We do not seem to be able to keep a > nightly OSX buildbot functioning, even one that builds once a week or so and > can be made available for major merges/releases would be better than none. > > If you have a lead to a modern OSX machine that can donate cycles to PyPy, > please respond or join on IRC at #pypy > > Thanks > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From matti.picus at gmail.com Fri May 22 15:36:22 2015 From: matti.picus at gmail.com (Matti Picus) Date: Fri, 22 May 2015 16:36:22 +0300 Subject: [pypy-dev] PyPy OSX buildbot needed In-Reply-To: References: <555F2C9B.80405@gmail.com> Message-ID: <555F30D6.8040002@gmail.com> That would be great! I would think you could hook in as a buildslave, then take the slave offline and bring it back online at your discretion. Once it is online, it would immediately begin building the backed-up scheduled builds, so you would have to go to http://buildbot.pypy.org/builders/pypy-c-jit-macosx-x86-64 and cancel the excess ones (if we don't find a nightly solution before then and leave the scheduler as currently set up). You seem to be able to build already, so you would just need to follow the instructions for setting up a slave here https://bitbucket.org/pypy/buildbot/src/98c105c0aaefe7392d6392ae5ba7f7c5e1348c81/README_BUILDSLAVE Matti On 22/05/15 16:26, ????? ??????? wrote: > Hello Matti! > I can do weekly builds and tests on weekends or on request (most of the time). > > From yorik.sar at gmail.com Sun May 24 20:13:18 2015 From: yorik.sar at gmail.com (Yuriy Taraday) Date: Sun, 24 May 2015 18:13:18 +0000 Subject: [pypy-dev] PyPy OSX buildbot needed In-Reply-To: <555F2C9B.80405@gmail.com> References: <555F2C9B.80405@gmail.com> Message-ID: On Fri, May 22, 2015 at 4:18 PM Matti Picus wrote: > If you have a lead to a modern OSX machine that can donate cycles to > PyPy, please respond or join on IRC at #pypy > Will a Hackintosh VM do? I've just finished installing one (with Niresh's Yosemite ISO) and it successfully built PyPy. I'm still to figure out how to better run tests though. It's on my home server that's up almost all time (may hang sometimes or reboot due to power fluctuations) and enough spare RAM (allocated 6G for it) and CPU cycles (packet forwarding will never load up all 4 cores of it). I know that it might be not good enough for producing proper releases, but for nightly tests it should be OK. -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Tue May 26 11:45:04 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 26 May 2015 17:45:04 +0800 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? Message-ID: We would like to monitor and log all external function calls from a class: when there the function call happen, we capture and log the frame. In python, there is the function PyEval_EvalFrameEx() we might add a callback inside. Is there a similar function in PyPy that would be called when the function call happens? -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue May 26 11:59:36 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 May 2015 11:59:36 +0200 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: what exactly are you doing? Are you modifying PyPy? Are you trying to access it from a C extension? On Tue, May 26, 2015 at 11:45 AM, Yicong Huang wrote: > We would like to monitor and log all external function calls from a class: > when there the function call happen, we capture and log the frame. > > In python, there is the function PyEval_EvalFrameEx() we might add a > callback inside. > Is there a similar function in PyPy that would be called when the function > call happens? > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From amauryfa at gmail.com Tue May 26 12:01:23 2015 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 26 May 2015 12:01:23 +0200 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: Did you consider the trace module? https://docs.python.org/2/library/trace.html 2015-05-26 11:45 GMT+02:00 Yicong Huang : > We would like to monitor and log all external function calls from a class: > when there the function call happen, we capture and log the frame. > > In python, there is the function PyEval_EvalFrameEx() we might add a > callback inside. > Is there a similar function in PyPy that would be called when the function > call happens? > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Tue May 26 12:31:14 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 26 May 2015 18:31:14 +0800 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: Yes, we are trying to modify PyPy to meet the need. Our plans are: 1. Find a similar function as "PyEval_EvalFrameEx()" in PyPy, and add a function logFunction() inside it. 2. In logFunction(), analyze the frame/callstack. If it is a function call from "Class A" to external function, log the function call. If it is possible, we might only need to log the same function call only once to minimize the perfomrance overhead. The problem is we've no ideas on which function in PyPy is similar to "PyEval_EvalFrameEx()" in Python. Or do you have any alternative good ideas to do this? On Tue, May 26, 2015 at 6:01 PM, Amaury Forgeot d'Arc wrote: > Did you consider the trace module? > > https://docs.python.org/2/library/trace.html > > 2015-05-26 11:45 GMT+02:00 Yicong Huang : > >> We would like to monitor and log all external function calls from a >> class: when there the function call happen, we capture and log the frame. >> >> In python, there is the function PyEval_EvalFrameEx() we might add a >> callback inside. >> Is there a similar function in PyPy that would be called when the >> function call happens? >> >> >> >> >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue May 26 12:32:58 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 May 2015 12:32:58 +0200 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: you will kill the JIT performance terribly if you do that. space.call or space.call_function (in pypy/objspace/std/objspace or pypy/interpreter/baseobjspace) are your candidates On Tue, May 26, 2015 at 12:31 PM, Yicong Huang wrote: > Yes, we are trying to modify PyPy to meet the need. > Our plans are: > 1. Find a similar function as "PyEval_EvalFrameEx()" in PyPy, and add a > function logFunction() inside it. > 2. In logFunction(), analyze the frame/callstack. If it is a function call > from "Class A" to external function, log the function call. If it is > possible, we might only need to log the same function call only once to > minimize the perfomrance overhead. > > The problem is we've no ideas on which function in PyPy is similar to > "PyEval_EvalFrameEx()" in Python. > Or do you have any alternative good ideas to do this? > > On Tue, May 26, 2015 at 6:01 PM, Amaury Forgeot d'Arc > wrote: >> >> Did you consider the trace module? >> >> https://docs.python.org/2/library/trace.html >> >> 2015-05-26 11:45 GMT+02:00 Yicong Huang : >>> >>> We would like to monitor and log all external function calls from a >>> class: when there the function call happen, we capture and log the frame. >>> >>> In python, there is the function PyEval_EvalFrameEx() we might add a >>> callback inside. >>> Is there a similar function in PyPy that would be called when the >>> function call happens? >>> >>> >>> >>> >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> https://mail.python.org/mailman/listinfo/pypy-dev >>> >> >> >> >> -- >> Amaury Forgeot d'Arc > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From hengha.mao at gmail.com Tue May 26 14:00:37 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 26 May 2015 20:00:37 +0800 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: Great thanks! I think you are right. We might kill the JIT perfomance by introducing such work. Another possible solution is we could do it at module level. Is there a function smilar as " PyMODINIT_FUNC()" in Pypy? Here is the plan: 1. When the code try to import an external module, we invoke an inserted function doLog(). 2. doLog() function record the module name and all module's public function. In this way, I think we could greatly reduce the perfomance overhead. On Tue, May 26, 2015 at 6:32 PM, Maciej Fijalkowski wrote: > you will kill the JIT performance terribly if you do that. space.call > or space.call_function (in pypy/objspace/std/objspace or > pypy/interpreter/baseobjspace) are your candidates > > On Tue, May 26, 2015 at 12:31 PM, Yicong Huang > wrote: > > Yes, we are trying to modify PyPy to meet the need. > > Our plans are: > > 1. Find a similar function as "PyEval_EvalFrameEx()" in PyPy, and add a > > function logFunction() inside it. > > 2. In logFunction(), analyze the frame/callstack. If it is a function > call > > from "Class A" to external function, log the function call. If it is > > possible, we might only need to log the same function call only once to > > minimize the perfomrance overhead. > > > > The problem is we've no ideas on which function in PyPy is similar to > > "PyEval_EvalFrameEx()" in Python. > > Or do you have any alternative good ideas to do this? > > > > On Tue, May 26, 2015 at 6:01 PM, Amaury Forgeot d'Arc < > amauryfa at gmail.com> > > wrote: > >> > >> Did you consider the trace module? > >> > >> https://docs.python.org/2/library/trace.html > >> > >> 2015-05-26 11:45 GMT+02:00 Yicong Huang : > >>> > >>> We would like to monitor and log all external function calls from a > >>> class: when there the function call happen, we capture and log the > frame. > >>> > >>> In python, there is the function PyEval_EvalFrameEx() we might add a > >>> callback inside. > >>> Is there a similar function in PyPy that would be called when the > >>> function call happens? > >>> > >>> > >>> > >>> > >>> _______________________________________________ > >>> pypy-dev mailing list > >>> pypy-dev at python.org > >>> https://mail.python.org/mailman/listinfo/pypy-dev > >>> > >> > >> > >> > >> -- > >> Amaury Forgeot d'Arc > > > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue May 26 14:04:07 2015 From: arigo at tunes.org (Armin Rigo) Date: Tue, 26 May 2015 14:04:07 +0200 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: Hi Yicong, On 26 May 2015 at 14:00, Yicong Huang wrote: > Here is the plan: > 1. When the code try to import an external module, we invoke an inserted > function doLog(). What do you mean by "an external module"? Is it a CPython C extension module imported via cpyext? I really think you want to look at sys.settrace() or sys.setprofile(). A bient?t, Armin. From hengha.mao at gmail.com Tue May 26 14:39:53 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Tue, 26 May 2015 20:39:53 +0800 Subject: [pypy-dev] Is there similar PyEval_EvalFrameEx() function in PyPy? In-Reply-To: References: Message-ID: "an external module" we would like to monitor includes C extension module and build-in module, e.g. fcntl, select. On Tue, May 26, 2015 at 8:04 PM, Armin Rigo wrote: > Hi Yicong, > > On 26 May 2015 at 14:00, Yicong Huang wrote: > > Here is the plan: > > 1. When the code try to import an external module, we invoke an inserted > > function doLog(). > > What do you mean by "an external module"? Is it a CPython C extension > module imported via cpyext? > > I really think you want to look at sys.settrace() or sys.setprofile(). > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Tue May 26 20:30:23 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 26 May 2015 13:30:23 -0500 Subject: [pypy-dev] What is the oldest version of CPython that can be used to build PyPy? Message-ID: ^ see subject -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronan.lamy at gmail.com Tue May 26 20:41:29 2015 From: ronan.lamy at gmail.com (Ronan Lamy) Date: Tue, 26 May 2015 19:41:29 +0100 Subject: [pypy-dev] What is the oldest version of CPython that can be used to build PyPy? In-Reply-To: References: Message-ID: <5564BE59.8070101@gmail.com> Le 26/05/15 19:30, Ryan Gonzalez a ?crit : > ^ see subject 2.7 From fijall at gmail.com Tue May 26 20:42:19 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 26 May 2015 20:42:19 +0200 Subject: [pypy-dev] What is the oldest version of CPython that can be used to build PyPy? In-Reply-To: References: Message-ID: I think we settled on 2.7 On Tue, May 26, 2015 at 8:30 PM, Ryan Gonzalez wrote: > ^ see subject > > -- > Ryan > [ERROR]: Your autotools build scripts are 200 lines longer than your > program. Something?s wrong. > http://kirbyfan64.github.io/ > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From rymg19 at gmail.com Tue May 26 20:58:45 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Tue, 26 May 2015 13:58:45 -0500 Subject: [pypy-dev] What is the oldest version of CPython that can be used to build PyPy? In-Reply-To: References: Message-ID: Ok. Thanks! On Tue, May 26, 2015 at 1:42 PM, Maciej Fijalkowski wrote: > I think we settled on 2.7 > > On Tue, May 26, 2015 at 8:30 PM, Ryan Gonzalez wrote: > > ^ see subject > > > > -- > > Ryan > > [ERROR]: Your autotools build scripts are 200 lines longer than your > > program. Something?s wrong. > > http://kirbyfan64.github.io/ > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue May 26 22:19:38 2015 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 26 May 2015 23:19:38 +0300 Subject: [pypy-dev] PyPy 2.6.0 release binaries available Message-ID: <5564D55A.6030101@gmail.com> The release packages are available on bitbucket https://bitbucket.org/pypy/pypy/downloads Please try them out, we have new buildslaves for win32, macosx and linux64. We would especially appreciate testing out the vmprof profiling features available on linux-64 Comments on the release notice are also welcome, extra points for a name incorporating "profiling" in it. http://pypy.readthedocs.org/en/release-2.6.x/release-2.6.0.html Matti From arigo at tunes.org Wed May 27 09:49:04 2015 From: arigo at tunes.org (Armin Rigo) Date: Wed, 27 May 2015 09:49:04 +0200 Subject: [pypy-dev] PyPy 2.6.0 release binaries available In-Reply-To: <5564D55A.6030101@gmail.com> References: <5564D55A.6030101@gmail.com> Message-ID: Hi Matti, On 26 May 2015 at 22:19, Matti Picus wrote: > The release packages are available on bitbucket > > https://bitbucket.org/pypy/pypy/downloads > > Please try them out, we have new buildslaves for win32, macosx and linux64. A problem with cffi has just been reported: https://bitbucket.org/cffi/cffi/issue/196/incorect-location-of-fficompile-d-py-file . It also occurs on CPython. If possible, I'd like the final 2.6.0 release to contain cffi 1.0.4, which I'm going to release now... Armin From phyo.arkarlwin at gmail.com Wed May 27 15:52:46 2015 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Wed, 27 May 2015 20:22:46 +0630 Subject: [pypy-dev] PyPy 2.6.0 release binaries available In-Reply-To: References: <5564D55A.6030101@gmail.com> Message-ID: Congrats! i love that pypy releases are getting faster and faster! i haven't even upgrade to 2.5.1 yet in my systems, 2.6 is relased. Time to try it out now and upgrade this one. On Wed, May 27, 2015 at 2:19 PM, Armin Rigo wrote: > Hi Matti, > > On 26 May 2015 at 22:19, Matti Picus wrote: > > The release packages are available on bitbucket > > > > https://bitbucket.org/pypy/pypy/downloads > > > > Please try them out, we have new buildslaves for win32, macosx and > linux64. > > A problem with cffi has just been reported: > > https://bitbucket.org/cffi/cffi/issue/196/incorect-location-of-fficompile-d-py-file > . It also occurs on CPython. If possible, I'd like the final 2.6.0 > release to contain cffi 1.0.4, which I'm going to release now... > > Armin > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Wed May 27 18:48:00 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 27 May 2015 11:48:00 -0500 Subject: [pypy-dev] PyPy 2.6.0 release binaries available In-Reply-To: References: <5564D55A.6030101@gmail.com> Message-ID: Not me! I just updated the Chocolatey package from 2.2 to 2.5.1 about 2 weeks ago! On May 27, 2015 8:52:46 AM CDT, Phyo Arkar wrote: >Congrats! i love that pypy releases are getting faster and faster! >i haven't even upgrade to 2.5.1 yet in my systems, 2.6 is relased. Time >to >try it out now and upgrade this one. > >On Wed, May 27, 2015 at 2:19 PM, Armin Rigo wrote: > >> Hi Matti, >> >> On 26 May 2015 at 22:19, Matti Picus wrote: >> > The release packages are available on bitbucket >> > >> > https://bitbucket.org/pypy/pypy/downloads >> > >> > Please try them out, we have new buildslaves for win32, macosx and >> linux64. >> >> A problem with cffi has just been reported: >> >> >https://bitbucket.org/cffi/cffi/issue/196/incorect-location-of-fficompile-d-py-file >> . It also occurs on CPython. If possible, I'd like the final 2.6.0 >> release to contain cffi 1.0.4, which I'm going to release now... >> >> Armin >> _______________________________________________ >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> > > >------------------------------------------------------------------------ > >_______________________________________________ >pypy-dev mailing list >pypy-dev at python.org >https://mail.python.org/mailman/listinfo/pypy-dev -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -------------- next part -------------- An HTML attachment was scrubbed... URL: From phyo.arkarlwin at gmail.com Wed May 27 19:04:28 2015 From: phyo.arkarlwin at gmail.com (Phyo Arkar) Date: Wed, 27 May 2015 23:34:28 +0630 Subject: [pypy-dev] PyPy 2.6.0 release binaries available In-Reply-To: References: <5564D55A.6030101@gmail.com> Message-ID: haha :D ! i am enjoying 2.5.0 so far , gonna wait for Portable-pypy to update to 2.6.0 , the binary releases have problem for my linux. On Wed, May 27, 2015 at 11:18 PM, Ryan Gonzalez wrote: > Not me! I just updated the Chocolatey package from 2.2 to 2.5.1 about 2 > weeks ago! > > > On May 27, 2015 8:52:46 AM CDT, Phyo Arkar > wrote: >> >> Congrats! i love that pypy releases are getting faster and faster! >> i haven't even upgrade to 2.5.1 yet in my systems, 2.6 is relased. Time >> to try it out now and upgrade this one. >> >> On Wed, May 27, 2015 at 2:19 PM, Armin Rigo wrote: >> >>> Hi Matti, >>> >>> On 26 May 2015 at 22:19, Matti Picus wrote: >>> > The release packages are available on bitbucket >>> > >>> > https://bitbucket.org/pypy/pypy/downloads >>> > >>> > Please try them out, we have new buildslaves for win32, macosx and >>> linux64. >>> >>> A problem with cffi has just been reported: >>> >>> https://bitbucket.org/cffi/cffi/issue/196/incorect-location-of-fficompile-d-py-file >>> . It also occurs on CPython. If possible, I'd like the final 2.6.0 >>> release to contain cffi 1.0.4, which I'm going to release now... >>> >>> Armin >>> _______________________________________________ >>> pypy-dev mailing list >>> pypy-dev at python.org >>> https://mail.python.org/mailman/listinfo/pypy-dev >>> >> >> ------------------------------ >> >> pypy-dev mailing list >> pypy-dev at python.org >> https://mail.python.org/mailman/listinfo/pypy-dev >> >> > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 28 12:31:09 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 28 May 2015 18:31:09 +0800 Subject: [pypy-dev] How to make a builtin pypy module visible? Message-ID: Hi, I tried to write a simple test module in pypy: 1. Make a new directory in pypy/modules, and put the code there. 2. Add the new module name in pypyoption.py and built the new libpypy-c.so The reason that I put the module in pypy/modules is I tried to make the module build into libpypy-c.so. :) After that, in pypy console, I am able to import the module and run module's function. However, when I tried to import the module in python script, I met the errors: pypy test.py Traceback (most recent call last): File "/app_main.py", line 75, in run_toplevel File "test.py", line 1, in import pypytest ImportError: No module named pypytest Any steps I missed to make the buitin module visible? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hengha.mao at gmail.com Thu May 28 13:46:39 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 28 May 2015 19:46:39 +0800 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Oh, I found out I should use the correct pypy(and .so) to run the python script. In previous test, "pypy test.py" point to the old one without the new module. Using the new built pypy, the result is good. On Thu, May 28, 2015 at 6:31 PM, Yicong Huang wrote: > Hi, > > I tried to write a simple test module in pypy: > 1. Make a new directory in pypy/modules, and put the code there. > 2. Add the new module name in pypyoption.py and built the new libpypy-c.so > > The reason that I put the module in pypy/modules is I tried to make the > module build into libpypy-c.so. :) > > After that, in pypy console, I am able to import the module and run > module's function. > However, when I tried to import the module in python script, I met the > errors: > > pypy test.py > > Traceback (most recent call last): > File "/app_main.py", line 75, in run_toplevel > File "test.py", line 1, in > import pypytest > ImportError: No module named pypytest > > Any steps I missed to make the buitin module visible? > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Thu May 28 13:50:20 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Thu, 28 May 2015 13:50:20 +0200 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: you likely don't want to compile pypy each time you change anything. Consider writing tests or using pyinteractive.py On Thu, May 28, 2015 at 1:46 PM, Yicong Huang wrote: > Oh, I found out I should use the correct pypy(and .so) to run the python > script. > In previous test, "pypy test.py" point to the old one without the new > module. > Using the new built pypy, the result is good. > > On Thu, May 28, 2015 at 6:31 PM, Yicong Huang wrote: >> >> Hi, >> >> I tried to write a simple test module in pypy: >> 1. Make a new directory in pypy/modules, and put the code there. >> 2. Add the new module name in pypyoption.py and built the new libpypy-c.so >> >> The reason that I put the module in pypy/modules is I tried to make the >> module build into libpypy-c.so. :) >> >> After that, in pypy console, I am able to import the module and run >> module's function. >> However, when I tried to import the module in python script, I met the >> errors: >> >> pypy test.py >> >> Traceback (most recent call last): >> File "/app_main.py", line 75, in run_toplevel >> File "test.py", line 1, in >> import pypytest >> ImportError: No module named pypytest >> >> Any steps I missed to make the buitin module visible? >> >> > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From hengha.mao at gmail.com Thu May 28 14:50:39 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 28 May 2015 20:50:39 +0800 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Great thanks for you advices! I tried to use the module for the lib in pypy/modules, but observed the following errors from the tests: Exception: object with a __call__ is not RPython: The function test_func is the function from the new built module. >From PyPy document, I saw that RPython is a subset of Python and more strict in syntaxt, but did not find details introduction on RPython. What is the possible casue to the error? And where we could find some syntax introductions on RPython. On Thu, May 28, 2015 at 7:50 PM, Maciej Fijalkowski wrote: > you likely don't want to compile pypy each time you change anything. > Consider writing tests or using pyinteractive.py > > On Thu, May 28, 2015 at 1:46 PM, Yicong Huang > wrote: > > Oh, I found out I should use the correct pypy(and .so) to run the python > > script. > > In previous test, "pypy test.py" point to the old one without the new > > module. > > Using the new built pypy, the result is good. > > > > On Thu, May 28, 2015 at 6:31 PM, Yicong Huang > wrote: > >> > >> Hi, > >> > >> I tried to write a simple test module in pypy: > >> 1. Make a new directory in pypy/modules, and put the code there. > >> 2. Add the new module name in pypyoption.py and built the new > libpypy-c.so > >> > >> The reason that I put the module in pypy/modules is I tried to make the > >> module build into libpypy-c.so. :) > >> > >> After that, in pypy console, I am able to import the module and run > >> module's function. > >> However, when I tried to import the module in python script, I met the > >> errors: > >> > >> pypy test.py > >> > >> Traceback (most recent call last): > >> File "/app_main.py", line 75, in run_toplevel > >> File "test.py", line 1, in > >> import pypytest > >> ImportError: No module named pypytest > >> > >> Any steps I missed to make the buitin module visible? > >> > >> > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 28 15:11:43 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 May 2015 15:11:43 +0200 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi Yicong, On 28 May 2015 at 14:50, Yicong Huang wrote: > What is the possible casue to the error? And where we could find some syntax > introductions on RPython. http://rpython.readthedocs.org/en/latest/rpython.html Note that this method of adding new modules is generally discouraged: http://pypy.readthedocs.org/en/latest/extending.html A bient?t, Armin. From hengha.mao at gmail.com Thu May 28 15:43:01 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 28 May 2015 21:43:01 +0800 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Thanks for the document! To my current understanding, RPython does not allow a class member function call. Is this correct? The module I wrote looks like this: pytest module: class a def func1: ... s = a() def func: s.func() And in another module to use pytest module, the code is import pytest .pytest.func() I think I should modify pytest module and remove class a: def func: ... Is this correct? On Thu, May 28, 2015 at 9:11 PM, Armin Rigo wrote: > Hi Yicong, > > On 28 May 2015 at 14:50, Yicong Huang wrote: > > What is the possible casue to the error? And where we could find some > syntax > > introductions on RPython. > > http://rpython.readthedocs.org/en/latest/rpython.html > > Note that this method of adding new modules is generally discouraged: > > http://pypy.readthedocs.org/en/latest/extending.html > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 28 15:57:53 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 May 2015 15:57:53 +0200 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi Yicong, If you really want to add an RPython module in PyPy, you should follow closely the example of another module. The first step is to add tests that run untranslated. Your module must be correct and tested *before* you attempt to translate it. Being correct RPython is not the first concern, as long as you follow the general guidelines shown by the other existing module. The last test to add (after all other tests) is a "test_ztranslation.py", which tests that your module is correct RPython code. We can discuss specific RPython issues you'd have at this point. If you only want to learn RPython, you could also try to write a small interpreter for a custom language. This is the path that other people take, and it allows them to choose which approach they prefer, e.g. if they don't care about untranslated tests. However, if you're working with PyPy, then you have to follow the rules and write tests, otherwise you're unlikely to get much help from any of us. (Also, it would help us get some motivation if you could successfully explain why you need an RPython module instead of, say, using CFFI.) A bient?t, Armin. From hengha.mao at gmail.com Thu May 28 17:09:09 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Thu, 28 May 2015 23:09:09 +0800 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi Armin, For the RPython module I added, I had added and passed test_ztranslation.py, and sucessfully built libpypy-c.so. And in pypy console, it worked correctly. The problem happened when I tried to use this RPython module for the existed RPython module in pypy/module. The test I failed was test_ztranslation.py. It reported the error: ... else: if hasattr(pyobj, '__call__'): msg = "object with a __call__ is not RPython" else: msg = "unexpected prebuilt constant" > raise Exception("%s: %r" % (msg, pyobj) E Exception: object with a __call__ is not RPython: The corresponding line is where I used my RPython module: myModule.testFunc() --------------------------------------------------------------------- And here are my thoughts why I need to use RPython module: I tried to implement a special lightweight sandbox or critical section to block certain builtin function. One user scenario might look like this: criticalSection.start() criticalSection.block('select.epoll') ...some code in critical section... criticcalSection.exit() Different from "usual sandbox", we would like to only block some customized builtin functions for a segment of the code. To achieve this purpose, here are my plans: 1. Write a builtin RPython module 'criticalsection', because I thought only builtin RPython module could be used for existed builtin RPython module 2. For the list of builtin functions that we might block, add the code in the begining of those functions, e.g. def epoll: if criticalsection.isInCriticalSection() and criticalsection.block('select.epoll') return None ... the original code... On Thu, May 28, 2015 at 9:57 PM, Armin Rigo wrote: > Hi Yicong, > > If you really want to add an RPython module in PyPy, you should follow > closely the example of another module. The first step is to add tests > that run untranslated. Your module must be correct and tested > *before* you attempt to translate it. Being correct RPython is not > the first concern, as long as you follow the general guidelines shown > by the other existing module. > > The last test to add (after all other tests) is a > "test_ztranslation.py", which tests that your module is correct > RPython code. We can discuss specific RPython issues you'd have at > this point. > > If you only want to learn RPython, you could also try to write a small > interpreter for a custom language. This is the path that other people > take, and it allows them to choose which approach they prefer, e.g. if > they don't care about untranslated tests. However, if you're working > with PyPy, then you have to follow the rules and write tests, > otherwise you're unlikely to get much help from any of us. > > (Also, it would help us get some motivation if you could successfully > explain why you need an RPython module instead of, say, using CFFI.) > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Thu May 28 17:41:56 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 May 2015 17:41:56 +0200 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi Yicong, On 28 May 2015 at 17:09, Yicong Huang wrote: > To achieve this purpose, here are my plans: > 1. Write a builtin RPython module 'criticalsection', because I thought only > builtin RPython module could be used for existed builtin RPython module > 2. For the list of builtin functions that we might block, add the code in > the begining of those functions, e.g. > > def epoll: > if criticalsection.isInCriticalSection() and > criticalsection.block('select.epoll') > return None > ... the original code... How about this pure Python solution, which would give the equivalent level of security (i.e., okish against naive code, but no hard security against a motivated attacker): def just_returns_none(*args, **kwds): return None def enter_critical_section(): socket.epoll = just_returns_none original_socket_epoll = socket.epoll def leave_criticial_section(): socket.epoll = original_socket_epoll A bient?t, Armin. From hengha.mao at gmail.com Thu May 28 18:11:45 2015 From: hengha.mao at gmail.com (Yicong Huang) Date: Fri, 29 May 2015 00:11:45 +0800 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi Armin, Thanks for the solution! I tired it, and found out a command "reload(socket)" could break it. So I might consider my previous method is more safer. The protection is at the function entry point. In addition, 'criticalsection' module and customized 'select' module are both build into libpypy-c.so. It might be more difficult to break. And for the mentioned RPython error: "object with a __call__ is not RPython", any advices? On Thu, May 28, 2015 at 11:41 PM, Armin Rigo wrote: > Hi Yicong, > > On 28 May 2015 at 17:09, Yicong Huang wrote: > > To achieve this purpose, here are my plans: > > 1. Write a builtin RPython module 'criticalsection', because I thought > only > > builtin RPython module could be used for existed builtin RPython module > > 2. For the list of builtin functions that we might block, add the code in > > the begining of those functions, e.g. > > > > def epoll: > > if criticalsection.isInCriticalSection() and > > criticalsection.block('select.epoll') > > return None > > ... the original code... > > How about this pure Python solution, which would give the equivalent > level of security (i.e., okish against naive code, but no hard > security against a motivated attacker): > > def just_returns_none(*args, **kwds): > return None > > def enter_critical_section(): > socket.epoll = just_returns_none > > original_socket_epoll = socket.epoll > > def leave_criticial_section(): > socket.epoll = original_socket_epoll > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ronan.lamy at gmail.com Thu May 28 18:30:52 2015 From: ronan.lamy at gmail.com (Ronan Lamy) Date: Thu, 28 May 2015 17:30:52 +0100 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: <556742BC.50309@gmail.com> Le 28/05/15 17:11, Yicong Huang a ?crit : > And for the mentioned RPython error: "object with a __call__ is not > RPython", any advices? In this case, it means that the translator sees a built-in function that it knows nothing about. My guess is that you're using an interpreter that you built yourself to run the translation. You should rather just use CPython to run the translation, it'll be less confusing. From arigo at tunes.org Thu May 28 18:33:56 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 May 2015 18:33:56 +0200 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi Yicong, On 28 May 2015 at 18:11, Yicong Huang wrote: > I tired it, and found out a command "reload(socket)" could break it. Then you need the same thing: patch "reload" from the "__builtin__" module. A bient?t, Armin. From arigo at tunes.org Thu May 28 19:35:05 2015 From: arigo at tunes.org (Armin Rigo) Date: Thu, 28 May 2015 19:35:05 +0200 Subject: [pypy-dev] How to make a builtin pypy module visible? In-Reply-To: References: Message-ID: Hi again, I'm suggesting to patch "reload" simply in order to fix the next obvious hole you found. By no means will you ever be able to obtain a truly secure environment in this way, but you can patch things as you find them. Depending on your goal it might be secure enough. For reference, you may be interested in reading more about this topic on the web. A starting point is: http://stackoverflow.com/questions/3068139/how-can-i-sandbox-python-in-pure-python , but use Google to find more. A bient?t, Armin. From rymg19 at gmail.com Sat May 30 03:32:27 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 29 May 2015 20:32:27 -0500 Subject: [pypy-dev] An overview of the RPython language Message-ID: I wrote a quick overview of the RPython language at http://kirbyfan64.github.io/posts/the-magic-of-rpython.html. Any suggestions for improvement would be appreciated. :) IMO, this is needed. Most RPython articles are insanely old (from 200x) or just cover the JIT (Andrew Brown's excellent tutorial). -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Sat May 30 13:18:36 2015 From: yury at shurup.com (Yury V. Zaytsev) Date: Sat, 30 May 2015 13:18:36 +0200 Subject: [pypy-dev] An overview of the RPython language In-Reply-To: References: Message-ID: <1432984716.17607.1404.camel@newpride> On Fri, 2015-05-29 at 20:32 -0500, Ryan Gonzalez wrote: > > IMO, this is needed. Most RPython articles are insanely old (from > 200x) or just cover the JIT (Andrew Brown's excellent tutorial). Very cool; I definitively enjoyed the reading a lot. Thanks! P.S. The formatting of the block after "The solution? You can use an assertion:" is messed up. -- Sincerely yours, Yury V. Zaytsev From rymg19 at gmail.com Sat May 30 18:28:51 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 30 May 2015 11:28:51 -0500 Subject: [pypy-dev] An overview of the RPython language In-Reply-To: <1432984716.17607.1404.camel@newpride> References: <1432984716.17607.1404.camel@newpride> Message-ID: <1B48E012-535E-4E6D-B956-D38479B33849@gmail.com> On May 30, 2015 6:18:36 AM CDT, "Yury V. Zaytsev" wrote: >On Fri, 2015-05-29 at 20:32 -0500, Ryan Gonzalez wrote: >> >> IMO, this is needed. Most RPython articles are insanely old (from >> 200x) or just cover the JIT (Andrew Brown's excellent tutorial). > >Very cool; I definitively enjoyed the reading a lot. Thanks! > >P.S. The formatting of the block after "The solution? You can use an >assertion:" is messed up. Thanks! I'll definitely fix that. :) -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From fijall at gmail.com Sun May 31 15:38:34 2015 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 31 May 2015 15:38:34 +0200 Subject: [pypy-dev] An overview of the RPython language In-Reply-To: <1B48E012-535E-4E6D-B956-D38479B33849@gmail.com> References: <1432984716.17607.1404.camel@newpride> <1B48E012-535E-4E6D-B956-D38479B33849@gmail.com> Message-ID: None in the target mens the default annotation of the entry point (list of strings) On Sat, May 30, 2015 at 6:28 PM, Ryan Gonzalez wrote: > > > On May 30, 2015 6:18:36 AM CDT, "Yury V. Zaytsev" wrote: >>On Fri, 2015-05-29 at 20:32 -0500, Ryan Gonzalez wrote: >>> >>> IMO, this is needed. Most RPython articles are insanely old (from >>> 200x) or just cover the JIT (Andrew Brown's excellent tutorial). >> >>Very cool; I definitively enjoyed the reading a lot. Thanks! >> >>P.S. The formatting of the block after "The solution? You can use an >>assertion:" is messed up. > > Thanks! I'll definitely fix that. :) > > -- > Sent from my Android device with K-9 Mail. Please excuse my brevity. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From rymg19 at gmail.com Sun May 31 23:40:42 2015 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 31 May 2015 16:40:42 -0500 Subject: [pypy-dev] An overview of the RPython language In-Reply-To: References: <1432984716.17607.1404.camel@newpride> <1B48E012-535E-4E6D-B956-D38479B33849@gmail.com> Message-ID: Thanks! I edited the post. On Sun, May 31, 2015 at 8:38 AM, Maciej Fijalkowski wrote: > None in the target mens the default annotation of the entry point > (list of strings) > > On Sat, May 30, 2015 at 6:28 PM, Ryan Gonzalez wrote: > > > > > > On May 30, 2015 6:18:36 AM CDT, "Yury V. Zaytsev" > wrote: > >>On Fri, 2015-05-29 at 20:32 -0500, Ryan Gonzalez wrote: > >>> > >>> IMO, this is needed. Most RPython articles are insanely old (from > >>> 200x) or just cover the JIT (Andrew Brown's excellent tutorial). > >> > >>Very cool; I definitively enjoyed the reading a lot. Thanks! > >> > >>P.S. The formatting of the block after "The solution? You can use an > >>assertion:" is messed up. > > > > Thanks! I'll definitely fix that. :) > > > > -- > > Sent from my Android device with K-9 Mail. Please excuse my brevity. > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > -- Ryan [ERROR]: Your autotools build scripts are 200 lines longer than your program. Something?s wrong. http://kirbyfan64.github.io/ -------------- next part -------------- An HTML attachment was scrubbed... URL: