From shubharamani at yahoo.com  Thu Dec  1 14:42:39 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 1 Dec 2016 19:42:39 +0000 (UTC)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
Message-ID: <652697406.4065596.1480621359881@mail.yahoo.com>

The documentation doesn't say much about compiling from source using Visual Studio and NO Cygwin.
?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161201/6f27f951/attachment.html>

From wlavrijsen at lbl.gov  Thu Dec  1 14:49:10 2016
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 1 Dec 2016 11:49:10 -0800 (PST)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <652697406.4065596.1480621359881@mail.yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
Message-ID: <alpine.LSU.2.11.1612011145390.3945@cp3-jyni>

Shubha,

> The documentation doesn't say much about compiling from source using
> Visual Studio and NO Cygwin.

probably mostly due to lack of access to Windows boxes.

But 'compiling' is an ambiguous term in building PyPy. I'll presume that
you're talking about the final step after translation here?

In $TMP/usession-<branch-name>-<seq.#>, you'll find testing_1/Makefile,
which is very straightforward and easily adapted.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From yury at shurup.com  Thu Dec  1 14:47:46 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Thu, 1 Dec 2016 20:47:46 +0100 (CET)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <652697406.4065596.1480621359881@mail.yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
Message-ID: <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>

On Thu, 1 Dec 2016, Shubha Ramani via pypy-dev wrote:

> The documentation doesn't say much about compiling from source using 
> Visual Studio and NO Cygwin.

What piece of documentation you are looking at more specifically?

My Windows builder is set up according to these instructions:

http://doc.pypy.org/en/latest/windows.html

I had to edit the vcvars whatever bat file, but otherwise it just works.

-- 
Sincerely yours,
Yury V. Zaytsev

From shubharamani at yahoo.com  Thu Dec  1 16:07:16 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 1 Dec 2016 21:07:16 +0000 (UTC)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
Message-ID: <1890732011.4146427.1480626436937@mail.yahoo.com>

Thank you for pointing me to the instructions. Does anyone know how to overcome this error ?
[translation:ERROR] CompilationError: CompilationError(err="""? ? ? ? platcheck_60.c? ? ? ? C:\Program Files (x86)\GnuWin32\include\openssl/ossl_typ.h(176) : error C2059: syntax error : 'type'? ? ? ? """)[translation] start debugger...> c:\users\sdramani\documents\projects\pypy\rpython\rtyper\tool\rffi_platform.py(847)configure_external_library()-> raise last_error(Pdb+)







?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Thursday, December 1, 2016 11:48 AM, Yury V. Zaytsev <yury at shurup.com> wrote:
 

 On Thu, 1 Dec 2016, Shubha Ramani via pypy-dev wrote:

> The documentation doesn't say much about compiling from source using 
> Visual Studio and NO Cygwin.

What piece of documentation you are looking at more specifically?

My Windows builder is set up according to these instructions:

http://doc.pypy.org/en/latest/windows.html

I had to edit the vcvars whatever bat file, but otherwise it just works.

-- 
Sincerely yours,
Yury V. Zaytsev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161201/d968c3bd/attachment.html>

From shubharamani at yahoo.com  Thu Dec  1 16:16:35 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 1 Dec 2016 21:16:35 +0000 (UTC)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <1890732011.4146427.1480626436937@mail.yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
 <1890732011.4146427.1480626436937@mail.yahoo.com>
Message-ID: <48682112.4153000.1480626995643@mail.yahoo.com>

OK QED. Stack Overflow came to the rescue:Openssl in windows issue

  
|  
|   
|   
|   |    |

   |

  |
|  
|   |  
Openssl in windows issue
 While I try to create lib files for windows.The following error came. Please help on this issue. CODE Explanati...  |   |

  |

  |

 
?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Thursday, December 1, 2016 1:08 PM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:
 

 Thank you for pointing me to the instructions. Does anyone know how to overcome this error ?
[translation:ERROR] CompilationError: CompilationError(err="""? ? ? ? platcheck_60.c? ? ? ? C:\Program Files (x86)\GnuWin32\include\openssl/ossl_typ.h(176) : error C2059: syntax error : 'type'? ? ? ? """)[translation] start debugger...> c:\users\sdramani\documents\projects\pypy\rpython\rtyper\tool\rffi_platform.py(847)configure_external_library()-> raise last_error(Pdb+)







?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Thursday, December 1, 2016 11:48 AM, Yury V. Zaytsev <yury at shurup.com> wrote:
 

 On Thu, 1 Dec 2016, Shubha Ramani via pypy-dev wrote:

> The documentation doesn't say much about compiling from source using 
> Visual Studio and NO Cygwin.

What piece of documentation you are looking at more specifically?

My Windows builder is set up according to these instructions:

http://doc.pypy.org/en/latest/windows.html

I had to edit the vcvars whatever bat file, but otherwise it just works.

-- 
Sincerely yours,
Yury V. Zaytsev


   
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161201/14f18b43/attachment-0001.html>

From shubharamani at yahoo.com  Thu Dec  1 21:33:55 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 1 Dec 2016 18:33:55 -0800
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <48682112.4153000.1480626995643@mail.yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
 <1890732011.4146427.1480626436937@mail.yahoo.com>
 <48682112.4153000.1480626995643@mail.yahoo.com>
Message-ID: <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>

I've nor been able to build OpenSSL 32-bit which is required for pypy windows build from source. Any suggestions? 

Sent from Shubha Ramani's iPhone 7

> On Dec 1, 2016, at 1:16 PM, Shubha Ramani <shubharamani at yahoo.com> wrote:
> 
> OK QED. Stack Overflow came to the rescue:
> Openssl in windows issue
> 
> Openssl in windows issue
> While I try to create lib files for windows.The following error came. Please help on this issue. CODE Explanati...
> 
>  
> Shubha D. Ramani
> shubharamani at gmail.com
> shubharamani at yahoo.com
> 
> 
> On Thursday, December 1, 2016 1:08 PM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:
> 
> 
> Thank you for pointing me to the instructions. Does anyone know how to overcome this error ?
> 
> [translation:ERROR] CompilationError: CompilationError(err="""
>         platcheck_60.c
>         C:\Program Files (x86)\GnuWin32\include\openssl/ossl_typ.h(176) : error C2059: syntax error : 'type'
>         """)
> [translation] start debugger...
> > c:\users\sdramani\documents\projects\pypy\rpython\rtyper\tool\rffi_platform.py(847)configure_external_library()
> -> raise last_error
> (Pdb+)
> 
> 
> 
> 
> 
> 
> 
> 
>  
> Shubha D. Ramani
> shubharamani at gmail.com
> shubharamani at yahoo.com
> 
> 
> On Thursday, December 1, 2016 11:48 AM, Yury V. Zaytsev <yury at shurup.com> wrote:
> 
> 
> On Thu, 1 Dec 2016, Shubha Ramani via pypy-dev wrote:
> 
> 
> > The documentation doesn't say much about compiling from source using 
> > Visual Studio and NO Cygwin.
> 
> 
> What piece of documentation you are looking at more specifically?
> 
> My Windows builder is set up according to these instructions:
> 
> http://doc.pypy.org/en/latest/windows.html
> 
> I had to edit the vcvars whatever bat file, but otherwise it just works.
> 
> -- 
> Sincerely yours,
> Yury V. Zaytsev
> 
> 
> 
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161201/32c7780a/attachment.html>

From wlavrijsen at lbl.gov  Fri Dec  2 00:17:59 2016
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 1 Dec 2016 21:17:59 -0800 (PST)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
 <1890732011.4146427.1480626436937@mail.yahoo.com>
 <48682112.4153000.1480626995643@mail.yahoo.com>
 <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>
Message-ID: <alpine.LSU.2.11.1612012115130.18303@cp3-jyni>

Shubha,

On Thursday 2016-12-01 18:33, Shubha Ramani via pypy-dev wrote:
> I've nor been able to build OpenSSL 32-bit which is required for pypy
> windows build from source. Any suggestions?

you should be able to get away with removing _ssl from the list of modules
to use. See: pypy/config/pypyoption.py+36 (variable 'working_modules').

For most work you want to trim down that list anyway, as doing so makes
the translation go a lot faster.

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From yury at shurup.com  Fri Dec  2 00:39:51 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 2 Dec 2016 06:39:51 +0100 (CET)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
 <1890732011.4146427.1480626436937@mail.yahoo.com>
 <48682112.4153000.1480626995643@mail.yahoo.com>
 <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>
Message-ID: <alpine.LRH.2.20.1612020638180.1584@vps.zaytsev.net>

On Thu, 1 Dec 2016, Shubha Ramani wrote:

> I've nor been able to build OpenSSL 32-bit which is required for pypy 
> windows build from source. Any suggestions?

... as the instructions suggest, start by using pre-compiled libraries and 
only then try to rebuild everything from scratch as you gain more 
experience.

-- 
Sincerely yours,
Yury V. Zaytsev

From shubharamani at yahoo.com  Fri Dec  2 12:53:22 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 2 Dec 2016 17:53:22 +0000 (UTC)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <alpine.LRH.2.20.1612020638180.1584@vps.zaytsev.net>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
 <1890732011.4146427.1480626436937@mail.yahoo.com>
 <48682112.4153000.1480626995643@mail.yahoo.com>
 <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>
 <alpine.LRH.2.20.1612020638180.1584@vps.zaytsev.net>
Message-ID: <488971852.4692001.1480701202180@mail.yahoo.com>

I give up on using SSL, pre-compiled or built from scratch [ IT JUST DOESN'T WORK].?I'm going to try what Wim suggested - remove SSLfrom the list of modules.
With all due respect has anyone tried building with the latest SSL binaries lately ?
I used pre-built binaries from here:https://slproweb.com/products/Win32OpenSSL.html

This is where I fail when building pypy every time:The pre-built binaries do not come with that file :?libeay32.lib

[translation:ERROR] CompilationError: CompilationError(err="""? ? ? ? LINK : fatal error LNK1181: cannot open input file 'libeay32.lib'? ? ? ? """)[translation] start debugger...> c:\users\sdramani\documents\projects\pypy\rpython\rtyper\tool\rffi_platform.py(847)configure_external_library()-> raise last_error(Pdb+)?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Thursday, December 1, 2016 9:40 PM, Yury V. Zaytsev <yury at shurup.com> wrote:
 

 On Thu, 1 Dec 2016, Shubha Ramani wrote:

> I've nor been able to build OpenSSL 32-bit which is required for pypy 
> windows build from source. Any suggestions?

... as the instructions suggest, start by using pre-compiled libraries and 
only then try to rebuild everything from scratch as you gain more 
experience.

-- 
Sincerely yours,
Yury V. Zaytsev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161202/16aecc85/attachment.html>

From wlavrijsen at lbl.gov  Fri Dec  2 14:29:17 2016
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Fri, 2 Dec 2016 11:29:17 -0800 (PST)
Subject: [pypy-dev] Can you compile pypy from source using Visual Studio
 no-cygwin ?
In-Reply-To: <488971852.4692001.1480701202180@mail.yahoo.com>
References: <652697406.4065596.1480621359881.ref@mail.yahoo.com>
 <652697406.4065596.1480621359881@mail.yahoo.com>
 <alpine.LRH.2.20.1612012045550.1584@vps.zaytsev.net>
 <1890732011.4146427.1480626436937@mail.yahoo.com>
 <48682112.4153000.1480626995643@mail.yahoo.com>
 <B82B9631-7A12-4200-AFFF-8797AEFB6530@yahoo.com>
 <alpine.LRH.2.20.1612020638180.1584@vps.zaytsev.net>
 <488971852.4692001.1480701202180@mail.yahoo.com>
Message-ID: <alpine.LSU.2.11.1612021117420.3945@cp3-jyni>

Shubna,

> I used pre-built binaries from here:
>  https://slproweb.com/products/Win32OpenSSL.html

the 2nd link on the wiki (at which I arrive from the openssl pages):

   https://indy.fulgan.com/SSL/

has zips with both libeay32.dll and ssleay32.dll. Also, the libs are named
explicitly in rpython/rlib/ropenssl.py+12, so if they are build/organized
differently by the distribution you use, you can change the names there.

(I don't know if the code really needs a .lib rather than .dll, but back
in the day one could be trivially created from the other by dumping the
symbols in a definition file for the linker.)

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From armin.rigo at gmail.com  Sat Dec  3 13:37:31 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 3 Dec 2016 19:37:31 +0100
Subject: [pypy-dev] Mercurial "general delta"
Message-ID: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>

Hi all,

A quick note for people that have a PyPy repo since years and are
using it like me without ever re-cloning from Bitbucket: the format of
modern mercurial repos is more compact (500MB instead of almost
900MB), but convertion of existing repos is not automatic.  The new
format is called General Delta.

If your directory ".hg/store/" is closer to 900 than 500 MB, try this:

hg clone -U --config format.generaldelta=1 --pull OLDREPO NEWREPO

You can then throw away the OLDREPO.  Or like me you can (1) make sure
OLDREPO is up-to-date with default without any local changes; (2) only
throw away OLDREPO/.hg and replace it with NEWREPO/.hg; (3) optionally
re-modify .hg/hgrc; (4) then do "hg update default" to resynchronize.

I noticed this because Bitbucket says the pypy repo is not General
Delta.  I don't know how to fix that, or if it really matters at all:
I guess newly cloned repos from Bitbucket will automatically use the
latest format anyway.


A bient?t,

Armin.

From armin.rigo at gmail.com  Sat Dec  3 13:45:06 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 3 Dec 2016 19:45:06 +0100
Subject: [pypy-dev] Mercurial "general delta"
In-Reply-To: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
References: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
Message-ID: <CAMSv6X3pxMHB_M13QBj6rx2zq5aQSqZU+BiOOZ3bv6J3QVFyrA@mail.gmail.com>

Hi,

On 3 December 2016 at 19:37, Armin Rigo <armin.rigo at gmail.com> wrote:
> format is called General Delta.

Note: this was introduced in Mercurial 3.5 and is the default for new
clones in Mercurial 3.7.


A bient?t,

Armin.

From fijall at gmail.com  Sun Dec  4 06:27:48 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 4 Dec 2016 13:27:48 +0200
Subject: [pypy-dev] Mercurial "general delta"
In-Reply-To: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
References: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
Message-ID: <CAK5idxRiB7oPHimRxnPSgtdrRTjrTwFAH=XA6xXnPuWqC3THwA@mail.gmail.com>

can we somehow make it new thing on the bb? that would mean download
gets smaller

On Sat, Dec 3, 2016 at 8:37 PM, Armin Rigo <armin.rigo at gmail.com> wrote:
> Hi all,
>
> A quick note for people that have a PyPy repo since years and are
> using it like me without ever re-cloning from Bitbucket: the format of
> modern mercurial repos is more compact (500MB instead of almost
> 900MB), but convertion of existing repos is not automatic.  The new
> format is called General Delta.
>
> If your directory ".hg/store/" is closer to 900 than 500 MB, try this:
>
> hg clone -U --config format.generaldelta=1 --pull OLDREPO NEWREPO
>
> You can then throw away the OLDREPO.  Or like me you can (1) make sure
> OLDREPO is up-to-date with default without any local changes; (2) only
> throw away OLDREPO/.hg and replace it with NEWREPO/.hg; (3) optionally
> re-modify .hg/hgrc; (4) then do "hg update default" to resynchronize.
>
> I noticed this because Bitbucket says the pypy repo is not General
> Delta.  I don't know how to fix that, or if it really matters at all:
> I guess newly cloned repos from Bitbucket will automatically use the
> latest format anyway.
>
>
> A bient?t,
>
> Armin.
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From armin.rigo at gmail.com  Sun Dec  4 06:28:55 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 4 Dec 2016 12:28:55 +0100
Subject: [pypy-dev] Mercurial "general delta"
In-Reply-To: <CAK5idxRiB7oPHimRxnPSgtdrRTjrTwFAH=XA6xXnPuWqC3THwA@mail.gmail.com>
References: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
 <CAK5idxRiB7oPHimRxnPSgtdrRTjrTwFAH=XA6xXnPuWqC3THwA@mail.gmail.com>
Message-ID: <CAMSv6X1vNmzQLxAdRra=Y4VRXZ5192yENtv79Q80sQDqa6EzQg@mail.gmail.com>

Hi Maciej,

On 4 December 2016 at 12:27, Maciej Fijalkowski <fijall at gmail.com> wrote:
> can we somehow make it new thing on the bb? that would mean download
> gets smaller

Is that true?


Armin

From fijall at gmail.com  Sun Dec  4 06:41:29 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 4 Dec 2016 13:41:29 +0200
Subject: [pypy-dev] Mercurial "general delta"
In-Reply-To: <CAMSv6X1vNmzQLxAdRra=Y4VRXZ5192yENtv79Q80sQDqa6EzQg@mail.gmail.com>
References: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
 <CAK5idxRiB7oPHimRxnPSgtdrRTjrTwFAH=XA6xXnPuWqC3THwA@mail.gmail.com>
 <CAMSv6X1vNmzQLxAdRra=Y4VRXZ5192yENtv79Q80sQDqa6EzQg@mail.gmail.com>
Message-ID: <CAK5idxSmOQgJ=Zb0AsregaEzt+p-ybKM7KK4hG9=NeqUZcpJSQ@mail.gmail.com>

I don't know :-)

let's see

On Sun, Dec 4, 2016 at 1:28 PM, Armin Rigo <armin.rigo at gmail.com> wrote:
> Hi Maciej,
>
> On 4 December 2016 at 12:27, Maciej Fijalkowski <fijall at gmail.com> wrote:
>> can we somehow make it new thing on the bb? that would mean download
>> gets smaller
>
> Is that true?
>
>
> Armin

From fijall at gmail.com  Sun Dec  4 10:35:29 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Sun, 4 Dec 2016 17:35:29 +0200
Subject: [pypy-dev] Mercurial "general delta"
In-Reply-To: <CAK5idxSmOQgJ=Zb0AsregaEzt+p-ybKM7KK4hG9=NeqUZcpJSQ@mail.gmail.com>
References: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
 <CAK5idxRiB7oPHimRxnPSgtdrRTjrTwFAH=XA6xXnPuWqC3THwA@mail.gmail.com>
 <CAMSv6X1vNmzQLxAdRra=Y4VRXZ5192yENtv79Q80sQDqa6EzQg@mail.gmail.com>
 <CAK5idxSmOQgJ=Zb0AsregaEzt+p-ybKM7KK4hG9=NeqUZcpJSQ@mail.gmail.com>
Message-ID: <CAK5idxQNrhgwkNyJSoBDtD2H3PouY3+f2mBGZcMZhbeccSJSew@mail.gmail.com>

as far as I understood the answer on #mercurial, it seems to be "no".
If the only effect is that it uses less space on bitbucket servers
then I presume it's not our problem

On Sun, Dec 4, 2016 at 1:41 PM, Maciej Fijalkowski <fijall at gmail.com> wrote:
> I don't know :-)
>
> let's see
>
> On Sun, Dec 4, 2016 at 1:28 PM, Armin Rigo <armin.rigo at gmail.com> wrote:
>> Hi Maciej,
>>
>> On 4 December 2016 at 12:27, Maciej Fijalkowski <fijall at gmail.com> wrote:
>>> can we somehow make it new thing on the bb? that would mean download
>>> gets smaller
>>
>> Is that true?
>>
>>
>> Armin

From opensource at ronnypfannschmidt.de  Sun Dec  4 16:08:51 2016
From: opensource at ronnypfannschmidt.de (Ronny Pfannschmidt)
Date: Sun, 4 Dec 2016 22:08:51 +0100
Subject: [pypy-dev] Mercurial "general delta"
In-Reply-To: <CAK5idxQNrhgwkNyJSoBDtD2H3PouY3+f2mBGZcMZhbeccSJSew@mail.gmail.com>
References: <CAMSv6X3_GUhMZHFgv33O86itDWpq4_yBQon5Oab2nJr+jUDdKg@mail.gmail.com>
 <CAK5idxRiB7oPHimRxnPSgtdrRTjrTwFAH=XA6xXnPuWqC3THwA@mail.gmail.com>
 <CAMSv6X1vNmzQLxAdRra=Y4VRXZ5192yENtv79Q80sQDqa6EzQg@mail.gmail.com>
 <CAK5idxSmOQgJ=Zb0AsregaEzt+p-ybKM7KK4hG9=NeqUZcpJSQ@mail.gmail.com>
 <CAK5idxQNrhgwkNyJSoBDtD2H3PouY3+f2mBGZcMZhbeccSJSew@mail.gmail.com>
Message-ID: <20161204220851.55dfdddd@ronny-ownpc>

Hi,

when i tried the same on the pypy repo a few years back,
the only potential solution would have been to do a complete
strip/removal of the server side history before re-uploading from a
generaldelta transformed repository. (im not sure if this still would
work)

if i recall correctly we didnt want to do it due to the need for a
*complete* strip on the server side


-- Ronny

Am Sun, 4 Dec 2016 17:35:29 +0200
schrieb Maciej Fijalkowski <fijall at gmail.com>:

> as far as I understood the answer on #mercurial, it seems to be "no".
> If the only effect is that it uses less space on bitbucket servers
> then I presume it's not our problem
> 
> On Sun, Dec 4, 2016 at 1:41 PM, Maciej Fijalkowski <fijall at gmail.com>
> wrote:
> > I don't know :-)
> >
> > let's see
> >
> > On Sun, Dec 4, 2016 at 1:28 PM, Armin Rigo <armin.rigo at gmail.com>
> > wrote:  
> >> Hi Maciej,
> >>
> >> On 4 December 2016 at 12:27, Maciej Fijalkowski <fijall at gmail.com>
> >> wrote:  
> >>> can we somehow make it new thing on the bb? that would mean
> >>> download gets smaller  
> >>
> >> Is that true?
> >>
> >>
> >> Armin  
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev


From shubharamani at yahoo.com  Mon Dec  5 13:36:05 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 18:36:05 +0000 (UTC)
Subject: [pypy-dev] with rpython is there a way to not write out the "c"
 files ?
References: <289503129.6166052.1480962965457.ref@mail.yahoo.com>
Message-ID: <289503129.6166052.1480962965457@mail.yahoo.com>

It takes a long time and I don't really care about the backend c implementation at the moment.Do I just do a -s as shown below ?
Command I'm running:
/opt/pypy/rpython$ ./bin/rpython -Ojit --jit-backend=auto ?./translator/goal/targetsimpleread.py
?
--annotate, -a ? ? ? ? ?Do type inference--no-annotate ? ? ? ? ? Don't do type inference--rtype, -t ? ? ? ? ? ? Do rtyping--no-rtype ? ? ? ? ? ? ?Don't do rtyping--pyjitpl ? ? ? ? ? ? ? JIT generation step--no-pyjitpl ? ? ? ? ? ?Don't JIT generation step--pyjittest ? ? ? ? ? ? JIT test with llgraph backend--no-jittest ? ? ? ? ? ?Don't JIT test with llgraph backend--backendopt ? ? ? ? ? ?Do backend optimizations--no-backendopt ? ? ? ? Don't do backend optimizations--source, -s ? ? ? ? ? ?Create source--no-source ? ? ? ? ? ? Don't create source--compile, -c ? ? ? ? ? Compile (default goal)--no-compile ? ? ? ? ? ?Don't compile--llinterpret ? ? ? ? ? Interpret the rtyped flow graphs--no-llinterpret ? ? ? ?Don't interpret the rtyped flow graphs
Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/dc71de54/attachment.html>

From rymg19 at gmail.com  Mon Dec  5 13:56:58 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Mon, 5 Dec 2016 12:56:58 -0600
Subject: [pypy-dev] with rpython is there a way to not write out the "c"
 files ?
In-Reply-To: <289503129.6166052.1480962965457@mail.yahoo.com>
References: <289503129.6166052.1480962965457.ref@mail.yahoo.com>
 <289503129.6166052.1480962965457@mail.yahoo.com>
Message-ID: <CAO41-mPyBABw--zyFegp-vrw9oNxq7Y=CieCW2mx2OXJ4MeqQA@mail.gmail.com>

Should be --no-source, but I have to wonder...what exactly are you trying
to do?

On Mon, Dec 5, 2016 at 12:36 PM, Shubha Ramani via pypy-dev <
pypy-dev at python.org> wrote:

> It takes a long time and I don't really care about the backend c
> implementation at the moment.
> Do I just do a -s as shown below ?
>
> Command I'm running:
>
> /opt/pypy/rpython$ ./bin/rpython -Ojit --jit-backend=auto
>  ./translator/goal/targetsimpleread.py
>
>
>
> --annotate, -a          Do type inference
> --no-annotate           Don't do type inference
> --rtype, -t             Do rtyping
> --no-rtype              Don't do rtyping
> --pyjitpl               JIT generation step
> --no-pyjitpl            Don't JIT generation step
> --pyjittest             JIT test with llgraph backend
> --no-jittest            Don't JIT test with llgraph backend
> --backendopt            Do backend optimizations
> --no-backendopt         Don't do backend optimizations
> --source, -s            Create source
> --no-source             Don't create source
> --compile, -c           Compile (default goal)
> --no-compile            Don't compile
> --llinterpret           Interpret the rtyped flow graphs
> --no-llinterpret        Don't interpret the rtyped flow graphs
>
>
> *Shubha D. Ramanishubharamani at gmail.com
> <shubharamani at gmail.com>shubharamani at yahoo.com <shubharamani at yahoo.com>*
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>


-- 
Ryan (????)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/f66984b4/attachment.html>

From shubharamani at yahoo.com  Mon Dec  5 14:06:50 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 19:06:50 +0000 (UTC)
Subject: [pypy-dev] with rpython is there a way to not write out the "c"
 files ?
In-Reply-To: <CAO41-mPyBABw--zyFegp-vrw9oNxq7Y=CieCW2mx2OXJ4MeqQA@mail.gmail.com>
References: <289503129.6166052.1480962965457.ref@mail.yahoo.com>
 <289503129.6166052.1480962965457@mail.yahoo.com>
 <CAO41-mPyBABw--zyFegp-vrw9oNxq7Y=CieCW2mx2OXJ4MeqQA@mail.gmail.com>
Message-ID: <211037870.6199877.1480964810145@mail.yahoo.com>

I'm trying to step through code and understand the interpreter. It turns out that --no-source is useless. I do need it because I need byte-codegeneration.
Shubha?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Monday, December 5, 2016 10:57 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
 

 Should be --no-source, but I have to wonder...what exactly are you trying to do?
On Mon, Dec 5, 2016 at 12:36 PM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:

It takes a long time and I don't really care about the backend c implementation at the moment.Do I just do a -s as shown below ?
Command I'm running:
/opt/pypy/rpython$ ./bin/rpython -Ojit --jit-backend=auto ?./translator/goal/ targetsimpleread.py
?
--annotate, -a ? ? ? ? ?Do type inference--no-annotate ? ? ? ? ? Don't do type inference--rtype, -t ? ? ? ? ? ? Do rtyping--no-rtype ? ? ? ? ? ? ?Don't do rtyping--pyjitpl ? ? ? ? ? ? ? JIT generation step--no-pyjitpl ? ? ? ? ? ?Don't JIT generation step--pyjittest ? ? ? ? ? ? JIT test with llgraph backend--no-jittest ? ? ? ? ? ?Don't JIT test with llgraph backend--backendopt ? ? ? ? ? ?Do backend optimizations--no-backendopt ? ? ? ? Don't do backend optimizations--source, -s ? ? ? ? ? ?Create source--no-source ? ? ? ? ? ? Don't create source--compile, -c ? ? ? ? ? Compile (default goal)--no-compile ? ? ? ? ? ?Don't compile--llinterpret ? ? ? ? ? Interpret the rtyped flow graphs--no-llinterpret ? ? ? ?Don't interpret the rtyped flow graphs
Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com
______________________________ _________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/ mailman/listinfo/pypy-dev





-- 
Ryan?(????)Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://kirbyfan64.github.io/?

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/7846c0aa/attachment-0001.html>

From shubharamani at yahoo.com  Mon Dec  5 14:07:33 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 19:07:33 +0000 (UTC)
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
Message-ID: <1065966981.6216124.1480964853621@mail.yahoo.com>


?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/078494a6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: blob.jpg
Type: image/png
Size: 80476 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/078494a6/attachment-0001.png>

From armin.rigo at gmail.com  Mon Dec  5 14:32:04 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 5 Dec 2016 20:32:04 +0100
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <1065966981.6216124.1480964853621@mail.yahoo.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
Message-ID: <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>

Hi Shubha,

You're trying to translate code that uses pdb.set_trace().  You can't do
that.  If you intend to debug that code, run it normally, don't try to
translate it.


Armin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/9190502b/attachment.html>

From armin.rigo at gmail.com  Mon Dec  5 14:39:48 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 5 Dec 2016 20:39:48 +0100
Subject: [pypy-dev] with rpython is there a way to not write out the "c"
 files ?
In-Reply-To: <211037870.6199877.1480964810145@mail.yahoo.com>
References: <289503129.6166052.1480962965457.ref@mail.yahoo.com>
 <289503129.6166052.1480962965457@mail.yahoo.com>
 <CAO41-mPyBABw--zyFegp-vrw9oNxq7Y=CieCW2mx2OXJ4MeqQA@mail.gmail.com>
 <211037870.6199877.1480964810145@mail.yahoo.com>
Message-ID: <CAMSv6X26WE6PiE3Uo_jjLUJ0GrdR9ZxVYvX9_3-hiVuKStEt8g@mail.gmail.com>

Hi again,

On 5 December 2016 at 20:06, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:
> I'm trying to step through code and understand the interpreter. It turns out
> that --no-source is useless. I do need it because I need byte-code
> generation.

Same answer, but I'll detail a bit more.  If you want to understand an
interpreter, run it on top of Python as usual; it's (also) a Python
program, so debug it like a Python program.  If instead you want to
understand the JIT and step through that, then see how the tests in
jit/metainterp/test/test_*.py are written---basically they translate
the interpreter (each test is its own 10-lines "interpreter") to an
in-memory format, and then run the untranslated JIT on that, so you
can step through the JIT.  There are also other ways other tests
elsewhere run, but the general idea is that *either* you debug, *or*
you translate, for every part that you're interested in (so for the
JIT, the tests translate the interpreter and then run the JIT
untranslated).  The GC works the same way, for example.


A bient?t,

Armin.

From shubharamani at yahoo.com  Mon Dec  5 14:40:47 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 19:40:47 +0000 (UTC)
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
Message-ID: <253986039.6214687.1480966847099@mail.yahoo.com>

OK. Please kindly give me a helpful nudge here.?I have read this :https://morepypy.blogspot.com/2011/04/tutorial-part-2-adding-jit.html

What I want to do is debug the translation to byte code process. How best to do that ? Use python print ?Since pdb is not going to work, I need some other way to debug. What can I do ?
Thanks,
Shubha?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Monday, December 5, 2016 11:32 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
 

 Hi Shubha,

You're trying to translate code that uses pdb.set_trace().? You can't do that.? If you intend to debug that code, run it normally, don't try to translate it.


Armin


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/afec0291/attachment.html>

From shubharamani at yahoo.com  Mon Dec  5 14:57:51 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 19:57:51 +0000 (UTC)
Subject: [pypy-dev] How to run the tests under
 /opt/pypy/rpython/jit/metainterp/test ?
References: <1086337147.6223112.1480967871829.ref@mail.yahoo.com>
Message-ID: <1086337147.6223112.1480967871829@mail.yahoo.com>

?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/2b5f8c4f/attachment.html>

From shubharamani at yahoo.com  Mon Dec  5 15:15:45 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 20:15:45 +0000 (UTC)
Subject: [pypy-dev] How to run the tests under
 /opt/pypy/rpython/jit/metainterp/test ?
In-Reply-To: <1086337147.6223112.1480967871829@mail.yahoo.com>
References: <1086337147.6223112.1480967871829.ref@mail.yahoo.com>
 <1086337147.6223112.1480967871829@mail.yahoo.com>
Message-ID: <309729909.6263266.1480968945489@mail.yahoo.com>

for instance,rpython/bin/rpython rpython/jit/metainterp/test/test_ajit.py

results in :

?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com 

    On Monday, December 5, 2016 11:57 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 ?Shubha D. Ramanishubharamani at gmail.com
shubharamani at yahoo.com

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/2342fab9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: blob.jpg
Type: image/png
Size: 90273 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/2342fab9/attachment-0001.png>

From armin.rigo at gmail.com  Mon Dec  5 15:30:36 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 5 Dec 2016 21:30:36 +0100
Subject: [pypy-dev] How to run the tests under
 /opt/pypy/rpython/jit/metainterp/test ?
In-Reply-To: <309729909.6263266.1480968945489@mail.yahoo.com>
References: <1086337147.6223112.1480967871829.ref@mail.yahoo.com>
 <1086337147.6223112.1480967871829@mail.yahoo.com>
 <309729909.6263266.1480968945489@mail.yahoo.com>
Message-ID: <CAMSv6X2FsUsYUvfgaT=RZZWJrXhwhz6oqxjg841c87Okq2YGkg@mail.gmail.com>

Hi,

pytest.py rpython/jit/metainterp/test/test_ajit.py

Also, can you please send plain-text e-mails without these huge
signatures?  Thanks!


A bient?t,

Armin.

From armin.rigo at gmail.com  Mon Dec  5 15:31:20 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 5 Dec 2016 21:31:20 +0100
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <253986039.6214687.1480966847099@mail.yahoo.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
 <253986039.6214687.1480966847099@mail.yahoo.com>
Message-ID: <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>

Hi,

On 5 December 2016 at 20:40, Shubha Ramani <shubharamani at yahoo.com> wrote:
> What I want to do is debug the translation to byte code process.

Sorry, I don't understand what you mean.  "Translation" here refers to
turning RPython code to C code.


Armin

From shubharamani at yahoo.com  Mon Dec  5 15:33:50 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 20:33:50 +0000 (UTC)
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
 <253986039.6214687.1480966847099@mail.yahoo.com>
 <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>
Message-ID: <334430117.6304169.1480970030812@mail.yahoo.com>

But in the process of turning RPython code into C code, there is a byte-code generationstep correct ?
Shubha 

    On Monday, December 5, 2016 12:32 PM, Armin Rigo <armin.rigo at gmail.com> wrote:
 

 Hi,

On 5 December 2016 at 20:40, Shubha Ramani <shubharamani at yahoo.com> wrote:
> What I want to do is debug the translation to byte code process.

Sorry, I don't understand what you mean.? "Translation" here refers to
turning RPython code to C code.


Armin


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/75f8a2fc/attachment.html>

From armin.rigo at gmail.com  Mon Dec  5 15:34:55 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 5 Dec 2016 21:34:55 +0100
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <334430117.6304169.1480970030812@mail.yahoo.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
 <253986039.6214687.1480966847099@mail.yahoo.com>
 <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>
 <334430117.6304169.1480970030812@mail.yahoo.com>
Message-ID: <CAMSv6X3XCiit6K2sYYMqky_5VDvyQCv0DG1E5+UVHe8SXkRm_g@mail.gmail.com>

Hi,

On 5 December 2016 at 21:33, Shubha Ramani <shubharamani at yahoo.com> wrote:
> But in the process of turning RPython code into C code, there is a byte-code
> generation step correct ?

No.

Armin

From shubharamani at yahoo.com  Mon Dec  5 15:40:17 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 20:40:17 +0000 (UTC)
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <CAMSv6X3XCiit6K2sYYMqky_5VDvyQCv0DG1E5+UVHe8SXkRm_g@mail.gmail.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
 <253986039.6214687.1480966847099@mail.yahoo.com>
 <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>
 <334430117.6304169.1480970030812@mail.yahoo.com>
 <CAMSv6X3XCiit6K2sYYMqky_5VDvyQCv0DG1E5+UVHe8SXkRm_g@mail.gmail.com>
Message-ID: <510844664.6265074.1480970417636@mail.yahoo.com>

Even if Jit is invoked during rpython translation ? Why not Armin ?I do see an assembler.py under rpython/jit/backend/x86



    On Monday, December 5, 2016 12:35 PM, Armin Rigo <armin.rigo at gmail.com> wrote:
 

 Hi,

On 5 December 2016 at 21:33, Shubha Ramani <shubharamani at yahoo.com> wrote:
> But in the process of turning RPython code into C code, there is a byte-code
> generation step correct ?

No.

Armin


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/a4418f6d/attachment.html>

From matti.picus at gmail.com  Mon Dec  5 15:42:03 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 5 Dec 2016 22:42:03 +0200
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <334430117.6304169.1480970030812@mail.yahoo.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
 <253986039.6214687.1480966847099@mail.yahoo.com>
 <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>
 <334430117.6304169.1480970030812@mail.yahoo.com>
Message-ID: <3608c721-e43a-4f7d-559c-efcd659f26c1@gmail.com>

On 05/12/16 22:33, Shubha Ramani via pypy-dev wrote:

> But in the process of turning RPython code into C code, there is a 
> byte-code generation
> step correct ?
>
> Shubha
>
You may want to carefully read the limited documentation available here
https://rpython.readthedocs.io/en/latest/architecture.html
especially
https://rpython.readthedocs.io/en/latest/translation.html

Matti

From matti.picus at gmail.com  Mon Dec  5 15:47:22 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Mon, 5 Dec 2016 22:47:22 +0200
Subject: [pypy-dev] please help. Why can't I do a pdb.set_trace() in
 llmodel.py at said line ?
In-Reply-To: <510844664.6265074.1480970417636@mail.yahoo.com>
References: <1065966981.6216124.1480964853621.ref@mail.yahoo.com>
 <1065966981.6216124.1480964853621@mail.yahoo.com>
 <CAMSv6X3mZL3t_vnqTGaWSpq5YGZRVA_Kmad98dz=jHYq0ByDOg@mail.gmail.com>
 <253986039.6214687.1480966847099@mail.yahoo.com>
 <CAMSv6X0dygNk==Gdu_FaZ2nt6jyxrbjuxFa_29AeprjB3WkjdA@mail.gmail.com>
 <334430117.6304169.1480970030812@mail.yahoo.com>
 <CAMSv6X3XCiit6K2sYYMqky_5VDvyQCv0DG1E5+UVHe8SXkRm_g@mail.gmail.com>
 <510844664.6265074.1480970417636@mail.yahoo.com>
Message-ID: <07892073-d06d-2621-c844-e36a431d1bed@gmail.com>

On 05/12/16 22:40, Shubha Ramani via pypy-dev wrote:

> Even if Jit is invoked during rpython translation ? Why not Armin ?
> I do see an assembler.py under rpython/jit/backend/x86
>
Note that this code is located under the jit directory. It is part of 
the JIT machinery,
which kicks in to compile hot loops to assembly at runtime.

The JIT is a tracing JIT, there is no general ahead-of-time compilation 
of code to assembler during translation
Matti

From shubharamani at yahoo.com  Mon Dec  5 15:48:24 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 5 Dec 2016 20:48:24 +0000 (UTC)
Subject: [pypy-dev] How to run the tests under
 /opt/pypy/rpython/jit/metainterp/test ?
In-Reply-To: <CAMSv6X2FsUsYUvfgaT=RZZWJrXhwhz6oqxjg841c87Okq2YGkg@mail.gmail.com>
References: <1086337147.6223112.1480967871829.ref@mail.yahoo.com>
 <1086337147.6223112.1480967871829@mail.yahoo.com>
 <309729909.6263266.1480968945489@mail.yahoo.com>
 <CAMSv6X2FsUsYUvfgaT=RZZWJrXhwhz6oqxjg841c87Okq2YGkg@mail.gmail.com>
Message-ID: <1529610038.20164.1480970904950@mail.yahoo.com>

Yes that worked.pytest.py rpython/jit/metainterp/test/test_ajit.py

Shubha 

    On Monday, December 5, 2016 12:31 PM, Armin Rigo <armin.rigo at gmail.com> wrote:
 

 Hi,

pytest.py rpython/jit/metainterp/test/test_ajit.py

Also, can you please send plain-text e-mails without these huge
signatures?? Thanks!


A bient?t,

Armin.

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161205/4e9a45d7/attachment.html>

From John.Zhang at anu.edu.au  Tue Dec  6 02:18:37 2016
From: John.Zhang at anu.edu.au (John Zhang)
Date: Tue, 6 Dec 2016 07:18:37 +0000
Subject: [pypy-dev] Request to make `makedev`, `major` and `minor` macros
Message-ID: <4809A89F-9D42-41C6-AC9E-71F04C8AD22C@anu.edu.au>

Hi all (Armin?),
I would like to request making the definitions of `rpython.rlib.rposix.c_makedev/c_major/c_minor` include explicit macro functions, i.e., passing `macro=_MACRO_ON_POSIX` to the `rffi.llexternal` function.
These functions are macros on both Linux and macOS (the comment above the code acknowledges this fact). Without explicitly saying they are macros, these functions can?t be called using rffi. The attached patch also adds another test in test_rposix.py, which exposes the problem.
The reason that they don?t currently affect the existing PyPy is because PyPy compiles to C anyway. But I?m working on another backend (Mu Micro VM), so I?d appreciate the ability to call these external functions from shared libraries directly. I think the macro wrapper generation is a great idea and its existing work is something that I can try to incorporate in my backend.
I would appreciate someone can patch the change (it?s a bit tricky on my end using git-remote-hg. :P ).
Regards,
John Zhang
------------------------------------------------------
John Zhang
Research Assistant
Programming Languages, Design & Implementation Division
Computer Systems Group
ANU College of Engineering & Computer Science
108 North Rd
The Australian National University
Acton ACT 2601
john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161206/97b9c1af/attachment.html>

From John.Zhang at anu.edu.au  Tue Dec  6 02:20:41 2016
From: John.Zhang at anu.edu.au (John Zhang)
Date: Tue, 6 Dec 2016 07:20:41 +0000
Subject: [pypy-dev] Request to make `makedev`, `major` and `minor` macros
In-Reply-To: <4809A89F-9D42-41C6-AC9E-71F04C8AD22C@anu.edu.au>
References: <4809A89F-9D42-41C6-AC9E-71F04C8AD22C@anu.edu.au>
Message-ID: <DE9C12B1-141C-439A-96CD-0EAFF3ECD39A@anu.edu.au>

Forgot the attachment, sorry!

------------------------------------------------------
John Zhang
Research Assistant
Programming Languages, Design & Implementation Division
Computer Systems Group
ANU College of Engineering & Computer Science
108 North Rd
The Australian National University
Acton ACT 2601
john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>





On 6 Dec. 2016, at 18:18, John Zhang <john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>> wrote:

Hi all (Armin?),
I would like to request making the definitions of `rpython.rlib.rposix.c_makedev/c_major/c_minor` include explicit macro functions, i.e., passing `macro=_MACRO_ON_POSIX` to the `rffi.llexternal` function.
These functions are macros on both Linux and macOS (the comment above the code acknowledges this fact). Without explicitly saying they are macros, these functions can?t be called using rffi. The attached patch also adds another test in test_rposix.py, which exposes the problem.
The reason that they don?t currently affect the existing PyPy is because PyPy compiles to C anyway. But I?m working on another backend (Mu Micro VM), so I?d appreciate the ability to call these external functions from shared libraries directly. I think the macro wrapper generation is a great idea and its existing work is something that I can try to incorporate in my backend.
I would appreciate someone can patch the change (it?s a bit tricky on my end using git-remote-hg. :P ).
Regards,
John Zhang
------------------------------------------------------
John Zhang
Research Assistant
Programming Languages, Design & Implementation Division
Computer Systems Group
ANU College of Engineering & Computer Science
108 North Rd
The Australian National University
Acton ACT 2601
john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>






-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161206/bd184591/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makedev_macro.patch
Type: application/octet-stream
Size: 1314 bytes
Desc: makedev_macro.patch
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161206/bd184591/attachment-0001.obj>

From John.Zhang at anu.edu.au  Mon Dec  5 21:06:09 2016
From: John.Zhang at anu.edu.au (John Zhang)
Date: Tue, 6 Dec 2016 02:06:09 +0000
Subject: [pypy-dev] RPython Darwin platform description so_prefixes empty
In-Reply-To: <CAMSv6X2RAVs4rM7wgCddYn=oiCBjQZ+x-v9KrrbWzk4ucD_Q+g@mail.gmail.com>
References: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au>
 <CAMSv6X2RAVs4rM7wgCddYn=oiCBjQZ+x-v9KrrbWzk4ucD_Q+g@mail.gmail.com>
Message-ID: <FF79FCAC-939C-49DD-8EDF-0DE41F113E6F@anu.edu.au>

Hi Armin,
The test `rpython.rtyper.lltypesystem.test.TestPlatform.test_prefix` illustrates the problem quite well. Currently this test is skipped on non-linux platforms. When that is turned off, the test fails on macOS X.
With `Darwin_x86.so_prefix == (??, )`, line ll2ctypes:1218 will fail, because it?s trying to load `c_file.dylib` rather than `libc_file.dylib`.
Shared libraries on macOS X should also have ?lib? prefix just as on linux.
According to man page of `ld`:
-lx This option tells the linker to search for libx.dylib or libx.a in the library search path. If string x is of the form y.o, then that file is searched for in the same places, but without prepending `lib' or appending `.a' or `.dylib' to the filename.
And on Apple developer website (https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/DynamicLibraryDesignGuidelines.html):
The filename of a dynamic library normally contains the library?s name with the lib prefix and the .dylib extension. For example, a library called Dynamo would have the filename libDynamo.dylib.

So I think it?s right to make a simple fix as suggested in the patch to include ?lib? in the so_prefix on Darwin platforms.

The attached is a suggested solution patch.

Regards,
John Zhang


------------------------------------------------------
John Zhang
Research Assistant
Programming Languages, Design & Implementation Division
Computer Systems Group
ANU College of Engineering & Computer Science
108 North Rd
The Australian National University
Acton ACT 2601
john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>





On 1 Dec. 2016, at 04:56, Armin Rigo <armin.rigo at gmail.com<mailto:armin.rigo at gmail.com>> wrote:

Hi John,

On 30 November 2016 at 04:14, John Zhang <John.Zhang at anu.edu.au<mailto:John.Zhang at anu.edu.au>> wrote:
`so_prefixes`. This is causing me some problems with not being able to find
my shared libraries on macOS X.
I?m wondering if that?s intentional, or just a mistake? It should be a
simple fix if it?s a mistake.

I don't know, and that's a good question.  But the first question is:
how does it work for PyPy on OS/X?  All (or most) tests pass there.
Can you pinpoint the difference?  If not, can you point us to your
version of the code, so that we can have a guess at what the
difference might be?


A bient?t,

Armin.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161206/448c38bf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_darwin_so_prefix.patch
Type: application/octet-stream
Size: 1953 bytes
Desc: fix_darwin_so_prefix.patch
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161206/448c38bf/attachment.obj>

From armin.rigo at gmail.com  Tue Dec  6 04:48:31 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Tue, 6 Dec 2016 10:48:31 +0100
Subject: [pypy-dev] RPython Darwin platform description so_prefixes empty
In-Reply-To: <FF79FCAC-939C-49DD-8EDF-0DE41F113E6F@anu.edu.au>
References: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au>
 <CAMSv6X2RAVs4rM7wgCddYn=oiCBjQZ+x-v9KrrbWzk4ucD_Q+g@mail.gmail.com>
 <FF79FCAC-939C-49DD-8EDF-0DE41F113E6F@anu.edu.au>
Message-ID: <CAMSv6X3+FjOXvgimPk=Au78WabKRBt2zDSxiH6S8pSxizs_Y6A@mail.gmail.com>

Hi John,

On 6 December 2016 at 03:06, John Zhang <John.Zhang at anu.edu.au> wrote:
> So I think it?s right to make a simple fix as suggested in the patch to
> include ?lib? in the so_prefix on Darwin platforms.

I think we can include 'lib' in so_prefix of all Posix platforms, is
that right?  It's already in linux, bsd and cygwin.  I still have no
clue why all our tests still pass on OS/X (apart from the skipped one
you found) but I guess that's a fine fix anyway.


A bient?t,

Armin.

From armin.rigo at gmail.com  Tue Dec  6 04:58:57 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Tue, 6 Dec 2016 10:58:57 +0100
Subject: [pypy-dev] Request to make `makedev`, `major` and `minor` macros
In-Reply-To: <DE9C12B1-141C-439A-96CD-0EAFF3ECD39A@anu.edu.au>
References: <4809A89F-9D42-41C6-AC9E-71F04C8AD22C@anu.edu.au>
 <DE9C12B1-141C-439A-96CD-0EAFF3ECD39A@anu.edu.au>
Message-ID: <CAMSv6X3-YUZUAX-6xbXrgw-PUJ-xh6J5ZuaNd6xuP7_tSKqKNg@mail.gmail.com>

Hi John,

On 6 December 2016 at 08:20, John Zhang <John.Zhang at anu.edu.au> wrote:
> I would like to request making the definitions of
> `rpython.rlib.rposix.c_makedev/c_major/c_minor` include explicit macro
> functions, i.e., passing `macro=_MACRO_ON_POSIX` to the `rffi.llexternal`
> function.

Checked in 08ec669ddab9.  Note that the 'macro' keyword is used inside
PyPy for a single purpose: to teach the JIT that it cannot directly
write a call to the macro.  Instead it writes a call to the wrapper
generated by rffi.py.  In this case this problem was fixed by marking
the functions like "def makedev(..):" with "@jit.dont_look_inside",
which is equivalent from the point of view of the JIT.  But I have now
removed these dont_look_inside and instead added "macro=True"
arguments, as it is more natural---and also indeed more useful to
non-standard backends.  Feel free to ask if you find other similar
cases!


A bient?t,

Armin.

From John.Zhang at anu.edu.au  Tue Dec  6 18:36:13 2016
From: John.Zhang at anu.edu.au (John Zhang)
Date: Tue, 6 Dec 2016 23:36:13 +0000
Subject: [pypy-dev] Request to make `makedev`, `major` and `minor` macros
In-Reply-To: <CAMSv6X3-YUZUAX-6xbXrgw-PUJ-xh6J5ZuaNd6xuP7_tSKqKNg@mail.gmail.com>
References: <4809A89F-9D42-41C6-AC9E-71F04C8AD22C@anu.edu.au>
 <DE9C12B1-141C-439A-96CD-0EAFF3ECD39A@anu.edu.au>
 <CAMSv6X3-YUZUAX-6xbXrgw-PUJ-xh6J5ZuaNd6xuP7_tSKqKNg@mail.gmail.com>
Message-ID: <2C63F373-F4D5-4F02-8B19-2BEEF41246D4@anu.edu.au>

Thanks Armin!
I will keep my eyes open for other cases and let you know. :)

Cheers,
John Zhang
------------------------------------------------------
John Zhang
Research Assistant
Programming Languages, Design & Implementation Division
Computer Systems Group
ANU College of Engineering & Computer Science
108 North Rd
The Australian National University
Acton ACT 2601
john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>





On 6 Dec. 2016, at 20:58, Armin Rigo <armin.rigo at gmail.com<mailto:armin.rigo at gmail.com>> wrote:

Hi John,

On 6 December 2016 at 08:20, John Zhang <John.Zhang at anu.edu.au<mailto:John.Zhang at anu.edu.au>> wrote:
I would like to request making the definitions of
`rpython.rlib.rposix.c_makedev/c_major/c_minor` include explicit macro
functions, i.e., passing `macro=_MACRO_ON_POSIX` to the `rffi.llexternal`
function.

Checked in 08ec669ddab9.  Note that the 'macro' keyword is used inside
PyPy for a single purpose: to teach the JIT that it cannot directly
write a call to the macro.  Instead it writes a call to the wrapper
generated by rffi.py.  In this case this problem was fixed by marking
the functions like "def makedev(..):" with "@jit.dont_look_inside",
which is equivalent from the point of view of the JIT.  But I have now
removed these dont_look_inside and instead added "macro=True"
arguments, as it is more natural---and also indeed more useful to
non-standard backends.  Feel free to ask if you find other similar
cases!


A bient?t,

Armin.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161206/255e5062/attachment.html>

From John.Zhang at anu.edu.au  Tue Dec  6 20:06:54 2016
From: John.Zhang at anu.edu.au (John Zhang)
Date: Wed, 7 Dec 2016 01:06:54 +0000
Subject: [pypy-dev] RPython Darwin platform description so_prefixes empty
In-Reply-To: <CAMSv6X3+FjOXvgimPk=Au78WabKRBt2zDSxiH6S8pSxizs_Y6A@mail.gmail.com>
References: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au>
 <CAMSv6X2RAVs4rM7wgCddYn=oiCBjQZ+x-v9KrrbWzk4ucD_Q+g@mail.gmail.com>
 <FF79FCAC-939C-49DD-8EDF-0DE41F113E6F@anu.edu.au>
 <CAMSv6X3+FjOXvgimPk=Au78WabKRBt2zDSxiH6S8pSxizs_Y6A@mail.gmail.com>
Message-ID: <73AED7CF-06D8-477D-B15D-7CE5D03A512F@anu.edu.au>

Hi Armin,
I think we can include 'lib' in so_prefix of all Posix platforms, is
that right?
I can?t seem to find a definitive answer for all POSIX platforms. But for all the ones that PyPy currently supports, I think the answer is true. So if you want to make the changes then I think it?s a reasonable assumption and a good idea.

I still have no
clue why all our tests still pass on OS/X
I think it?s because `rpython.translator.platform.Platform._finish_linking` function doesn?t include `so_prefixes` on the platform in the first place, so for tests like `test_lib_on_libpaths` will generate a shared library called `c_file.dylib`. Then in `get_ctypes_callable` function it prepends the library name with the currently empty prefix on Darwin platform, it can find the shared library file with the constructed absolute path. So the proposed fix will still work because it includes both ?lib? and empty prefix.
On the other hand, I wonder if it?s also reasonable to modify the `_finish_linking` function to include `platform.so_prefix[0]` to the generated shared library name (see the attached patch).
Regards,
John Zhang

------------------------------------------------------
John Zhang
Research Assistant
Programming Languages, Design & Implementation Division
Computer Systems Group
ANU College of Engineering & Computer Science
108 North Rd
The Australian National University
Acton ACT 2601
john.zhang at anu.edu.au<mailto:john.zhang at anu.edu.au>





On 6 Dec. 2016, at 20:48, Armin Rigo <armin.rigo at gmail.com<mailto:armin.rigo at gmail.com>> wrote:

Hi John,

On 6 December 2016 at 03:06, John Zhang <John.Zhang at anu.edu.au<mailto:John.Zhang at anu.edu.au>> wrote:
So I think it?s right to make a simple fix as suggested in the patch to
include ?lib? in the so_prefix on Darwin platforms.

I think we can include 'lib' in so_prefix of all Posix platforms, is
that right?  It's already in linux, bsd and cygwin.  I still have no
clue why all our tests still pass on OS/X (apart from the skipped one
you found) but I guess that's a fine fix anyway.


A bient?t,

Armin.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161207/3868188b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: _finish_linking_so_prefix.patch
Type: application/octet-stream
Size: 703 bytes
Desc: _finish_linking_so_prefix.patch
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161207/3868188b/attachment-0001.obj>

From shubharamani at yahoo.com  Thu Dec  8 16:49:39 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 8 Dec 2016 21:49:39 +0000 (UTC)
Subject: [pypy-dev] why doesn't this code dump jit debug info ?
References: <1142488916.651298.1481233779504.ref@mail.yahoo.com>
Message-ID: <1142488916.651298.1481233779504@mail.yahoo.com>

import py ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??import binary_trees ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??import os ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??from rpython.rlib.rjitlog import rjitlog ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??def test_binary_trees(): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? file = ('./jit.log') ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? fileno = os.open(file, os.O_WRONLY | os.O_CREAT) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? enable_jitlog = lambda: rjitlog.enable_jitlog(fileno) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? os.system("./binary_trees.py 100 100") ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ?

running it as ?/opt/pypy/pytest.py test_shubha.py
The log gets created but it is empty?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161208/a1683d48/attachment.html>

From dmalcolm at redhat.com  Thu Dec  8 17:12:35 2016
From: dmalcolm at redhat.com (David Malcolm)
Date: Thu, 08 Dec 2016 17:12:35 -0500
Subject: [pypy-dev] why doesn't this code dump jit debug info ?
In-Reply-To: <1142488916.651298.1481233779504@mail.yahoo.com>
References: <1142488916.651298.1481233779504.ref@mail.yahoo.com>
 <1142488916.651298.1481233779504@mail.yahoo.com>
Message-ID: <1481235155.16607.14.camel@redhat.com>

On Thu, 2016-12-08 at 21:49 +0000, Shubha Ramani via pypy-dev wrote:
> import py                                                            
>                                 
> import binary_trees                                                  
>                                 
> import os                                                            
>                                 
> from rpython.rlib.rjitlog import rjitlog                             
>                                  
>                                                                      
>                                 
> def test_binary_trees():                                             
>                                  
>     file = ('./jit.log')                                             
>                                  
>     fileno = os.open(file, os.O_WRONLY | os.O_CREAT)                 
>                                  
>     enable_jitlog = lambda: rjitlog.enable_jitlog(fileno)            
>                                 
>     os.system("./binary_trees.py 100 100")                           
>                                  
>                                                                      
>                                 
>                    



> running it as  /opt/pypy/pytest.py test_shubha.py
> 
> The log gets created but it is empty 

Firstly, are you actually calling test_binary_trees?  If not, you're
just importing some modules, and defining a function, then exiting.

Secondly: os.system will run the binary_trees.py in a separate process,
invoking it via whatever is in the shebang line.  That won't inherit
any of the jitlog settings you've done in this process (and if it's
#!/usr/bin/python, it could be CPython you're invoking).


Hope this is helpful
Dave

From shubharamani at yahoo.com  Thu Dec  8 18:21:40 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 8 Dec 2016 23:21:40 +0000 (UTC)
Subject: [pypy-dev] why doesn't this code dump jit debug info ?
In-Reply-To: <1481235155.16607.14.camel@redhat.com>
References: <1142488916.651298.1481233779504.ref@mail.yahoo.com>
 <1142488916.651298.1481233779504@mail.yahoo.com>
 <1481235155.16607.14.camel@redhat.com>
Message-ID: <1812090195.727740.1481239300287@mail.yahoo.com>

Yes I finally figured it out (what you are saying David). ?I did get a binary jit log dump intoa file finally. How do you parse it ? 

    On Thursday, December 8, 2016 2:12 PM, David Malcolm <dmalcolm at redhat.com> wrote:
 

 On Thu, 2016-12-08 at 21:49 +0000, Shubha Ramani via pypy-dev wrote:
> import py? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
> import binary_trees? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
> import os? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
> from rpython.rlib.rjitlog import rjitlog? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
> def test_binary_trees():? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? file = ('./jit.log')? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? fileno = os.open(file, os.O_WRONLY | os.O_CREAT)? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? enable_jitlog = lambda: rjitlog.enable_jitlog(fileno)? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? os.system("./binary_trees.py 100 100")? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? 
>? ? ? ? ? ? ? ? ? ? 



> running it as? /opt/pypy/pytest.py test_shubha.py
> 
> The log gets created but it is empty 

Firstly, are you actually calling test_binary_trees?? If not, you're
just importing some modules, and defining a function, then exiting.

Secondly: os.system will run the binary_trees.py in a separate process,
invoking it via whatever is in the shebang line.? That won't inherit
any of the jitlog settings you've done in this process (and if it's
#!/usr/bin/python, it could be CPython you're invoking).


Hope this is helpful
Dave


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161208/91559d90/attachment-0001.html>

From shubharamani at yahoo.com  Thu Dec  8 20:00:40 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 01:00:40 +0000 (UTC)
Subject: [pypy-dev] int_py_mod_nonnegargs(): one arg is negative
 LLAssertFailure
References: <34511151.730350.1481245240646.ref@mail.yahoo.com>
Message-ID: <34511151.730350.1481245240646@mail.yahoo.com>

Why am I getting this error ?pidigits.py runs fine outside of the pytest environment.
Please see attached files
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/e84b621e/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pidigits.py
Type: text/x-python
Size: 742 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/e84b621e/attachment-0002.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jitlog_test.py
Type: text/x-python
Size: 4785 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/e84b621e/attachment-0003.py>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: error.log
Type: text/x-log
Size: 30398 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/e84b621e/attachment-0001.bin>

From shubharamani at yahoo.com  Thu Dec  8 20:03:41 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 01:03:41 +0000 (UTC)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
Message-ID: <70460416.752349.1481245421593@mail.yahoo.com>

I successfully created a file containing binary data, but don't know what to do with it.I produced it using test?/opt/pypy/rpython/jit/backend/x86/test/test_jitlog.py
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/1ee427b0/attachment.html>

From armin.rigo at gmail.com  Fri Dec  9 03:45:33 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 9 Dec 2016 09:45:33 +0100
Subject: [pypy-dev] int_py_mod_nonnegargs(): one arg is negative
 LLAssertFailure
In-Reply-To: <34511151.730350.1481245240646@mail.yahoo.com>
References: <34511151.730350.1481245240646.ref@mail.yahoo.com>
 <34511151.730350.1481245240646@mail.yahoo.com>
Message-ID: <CAMSv6X1VeND=EgW=DL7PCtnPq7d5DDjwoeYsfhgv73Vf00LDtg@mail.gmail.com>

Hi,

On 9 December 2016 at 02:00, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:
> Why am I getting this error ?
> pidigits.py runs fine outside of the pytest environment.

I'm getting:

    from rpython.jit.metainterp.test.support import LLJitMixin,X86JitMixin
E   ImportError: cannot import name X86JitMixin

After hacking at that until the test runs (replacing it with
LLJitMixin), then I see the problem.  It's likely caused by integer
overflow.  Your f() function computes the digits of pi, but it's not
valid RPython because integers overflow the size of C 'long'.  An
internal test then fails because although '(n*3 + a) % d' was
supposedly proven to occur on known-nonnegative arguments, the value
did overflow and is now negative.  Copying the same algorithm in C
with the type 'long' everywhere would be similarly undefined C code.


Armin

From planrichi at gmail.com  Fri Dec  9 04:19:02 2016
From: planrichi at gmail.com (Richard Plangger)
Date: Fri, 9 Dec 2016 10:19:02 +0100
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <70460416.752349.1481245421593@mail.yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
Message-ID: <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>

Hi,

the jit log currently is only functional with the vmprof platform.
There was this idea of a command line tool to inspect, but it is not yet
finished.

Unsure if the test generated file can be uploaded, but you can try:

$ pip install vmprof
$ python -m jitlog --upload <your-file-file-name>

The output of the last program will upload the logfile to vmprof.com and
you can inspect it by clicking on the link it prints at the end.

Try the following to get a log out of pypy for your program:

(pypy-env) $ pip install vmprof
(pypy-env) $ pypy -m vmprof --jitlog yourpythonprogram.py

Cheers,
Richard

On 12/09/2016 02:03 AM, Shubha Ramani via pypy-dev wrote:
> I successfully created a file containing binary data, but don't know
> what to do with it.
> I produced it using
> test /opt/pypy/rpython/jit/backend/x86/test/test_jitlog.py
> 
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/0b99b82b/attachment.sig>

From shubharamani at yahoo.com  Fri Dec  9 08:38:54 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 05:38:54 -0800
Subject: [pypy-dev] int_py_mod_nonnegargs(): one arg is negative
 LLAssertFailure
In-Reply-To: <CAMSv6X1VeND=EgW=DL7PCtnPq7d5DDjwoeYsfhgv73Vf00LDtg@mail.gmail.com>
References: <34511151.730350.1481245240646.ref@mail.yahoo.com>
 <34511151.730350.1481245240646@mail.yahoo.com>
 <CAMSv6X1VeND=EgW=DL7PCtnPq7d5DDjwoeYsfhgv73Vf00LDtg@mail.gmail.com>
Message-ID: <1989E686-4809-4DAB-ABB0-F5A68D2873CD@yahoo.com>

Interesting. Thank You !


> On Dec 9, 2016, at 12:45 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
> 
> Hi,
> 
> On 9 December 2016 at 02:00, Shubha Ramani via pypy-dev
> <pypy-dev at python.org> wrote:
>> Why am I getting this error ?
>> pidigits.py runs fine outside of the pytest environment.
> 
> I'm getting:
> 
>    from rpython.jit.metainterp.test.support import LLJitMixin,X86JitMixin
> E   ImportError: cannot import name X86JitMixin
> 
> After hacking at that until the test runs (replacing it with
> LLJitMixin), then I see the problem.  It's likely caused by integer
> overflow.  Your f() function computes the digits of pi, but it's not
> valid RPython because integers overflow the size of C 'long'.  An
> internal test then fails because although '(n*3 + a) % d' was
> supposedly proven to occur on known-nonnegative arguments, the value
> did overflow and is now negative.  Copying the same algorithm in C
> with the type 'long' everywhere would be similarly undefined C code.
> 
> 
> Armin

From shubharamani at yahoo.com  Fri Dec  9 08:43:14 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 05:43:14 -0800
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
Message-ID: <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>

Is there a way to get Jitlog WITHOUT running it under test and calling metainterp functions ? What if I just want to run some open source benchmark ? What if I don't want to add JItdriver and merge_point to the code and run it "as is" non rpython - can I still dump jitlog ?

Sent from Shubha Ramani's iPhone 7

> On Dec 9, 2016, at 1:19 AM, Richard Plangger <planrichi at gmail.com> wrote:
> 
> Hi,
> 
> the jit log currently is only functional with the vmprof platform.
> There was this idea of a command line tool to inspect, but it is not yet
> finished.
> 
> Unsure if the test generated file can be uploaded, but you can try:
> 
> $ pip install vmprof
> $ python -m jitlog --upload <your-file-file-name>
> 
> The output of the last program will upload the logfile to vmprof.com and
> you can inspect it by clicking on the link it prints at the end.
> 
> Try the following to get a log out of pypy for your program:
> 
> (pypy-env) $ pip install vmprof
> (pypy-env) $ pypy -m vmprof --jitlog yourpythonprogram.py
> 
> Cheers,
> Richard
> 
>> On 12/09/2016 02:03 AM, Shubha Ramani via pypy-dev wrote:
>> I successfully created a file containing binary data, but don't know
>> what to do with it.
>> I produced it using
>> test /opt/pypy/rpython/jit/backend/x86/test/test_jitlog.py
>> 
>> 
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>> 
> 


From planrichi at gmail.com  Fri Dec  9 09:29:46 2016
From: planrichi at gmail.com (Richard Plangger)
Date: Fri, 9 Dec 2016 15:29:46 +0100
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
Message-ID: <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>

Jitlog is data generated by the JIT compiler. Without a jitdriver, it
will not be able to JIT compile any code, thus you cannot obtain that
file. If you do not have a rpython program, you cannot really generate
the jitlog easily.

I'm unsure what you want to achieve, if you give me more information
what you mean by 'run some open source benchmark' or 'I don't want to
add JItdriver'?

Cheers,
Richard

On 12/09/2016 02:43 PM, Shubha Ramani wrote:
> Is there a way to get Jitlog WITHOUT running it under test and calling metainterp functions ? What if I just want to run some open source benchmark ? What if I don't want to add JItdriver and merge_point to the code and run it "as is" non rpython - can I still dump jitlog ?
> 
> Sent from Shubha Ramani's iPhone 7
> 
>> On Dec 9, 2016, at 1:19 AM, Richard Plangger <planrichi at gmail.com> wrote:
>>
>> Hi,
>>
>> the jit log currently is only functional with the vmprof platform.
>> There was this idea of a command line tool to inspect, but it is not yet
>> finished.
>>
>> Unsure if the test generated file can be uploaded, but you can try:
>>
>> $ pip install vmprof
>> $ python -m jitlog --upload <your-file-file-name>
>>
>> The output of the last program will upload the logfile to vmprof.com and
>> you can inspect it by clicking on the link it prints at the end.
>>
>> Try the following to get a log out of pypy for your program:
>>
>> (pypy-env) $ pip install vmprof
>> (pypy-env) $ pypy -m vmprof --jitlog yourpythonprogram.py
>>
>> Cheers,
>> Richard
>>
>>> On 12/09/2016 02:03 AM, Shubha Ramani via pypy-dev wrote:
>>> I successfully created a file containing binary data, but don't know
>>> what to do with it.
>>> I produced it using
>>> test /opt/pypy/rpython/jit/backend/x86/test/test_jitlog.py
>>>
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/1ceb6409/attachment.sig>

From shubharamani at yahoo.com  Fri Dec  9 09:40:16 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 14:40:16 +0000 (UTC)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
Message-ID: <360320017.1062071.1481294417004@mail.yahoo.com>

I'm building a tool which attempts to map Jit ByteCodes back to the actual function which was called at the Python level.Therefore I need to map the Jit ByteCodes back to the actual function address - the physical address not the virtual.It seems like to build this tool I have to get the rpython version of the open source benchmark - which is not an easy task.I would like to take any old Python application out in the wild and instrument it to dump jitcodes using rpython as an interpreter.?Is this possible ? 

    On Friday, December 9, 2016 6:29 AM, Richard Plangger <planrichi at gmail.com> wrote:
 

 Jitlog is data generated by the JIT compiler. Without a jitdriver, it
will not be able to JIT compile any code, thus you cannot obtain that
file. If you do not have a rpython program, you cannot really generate
the jitlog easily.

I'm unsure what you want to achieve, if you give me more information
what you mean by 'run some open source benchmark' or 'I don't want to
add JItdriver'?

Cheers,
Richard

On 12/09/2016 02:43 PM, Shubha Ramani wrote:
> Is there a way to get Jitlog WITHOUT running it under test and calling metainterp functions ? What if I just want to run some open source benchmark ? What if I don't want to add JItdriver and merge_point to the code and run it "as is" non rpython - can I still dump jitlog ?
> 
> Sent from Shubha Ramani's iPhone 7
> 
>> On Dec 9, 2016, at 1:19 AM, Richard Plangger <planrichi at gmail.com> wrote:
>>
>> Hi,
>>
>> the jit log currently is only functional with the vmprof platform.
>> There was this idea of a command line tool to inspect, but it is not yet
>> finished.
>>
>> Unsure if the test generated file can be uploaded, but you can try:
>>
>> $ pip install vmprof
>> $ python -m jitlog --upload <your-file-file-name>
>>
>> The output of the last program will upload the logfile to vmprof.com and
>> you can inspect it by clicking on the link it prints at the end.
>>
>> Try the following to get a log out of pypy for your program:
>>
>> (pypy-env) $ pip install vmprof
>> (pypy-env) $ pypy -m vmprof --jitlog yourpythonprogram.py
>>
>> Cheers,
>> Richard
>>
>>> On 12/09/2016 02:03 AM, Shubha Ramani via pypy-dev wrote:
>>> I successfully created a file containing binary data, but don't know
>>> what to do with it.
>>> I produced it using
>>> test /opt/pypy/rpython/jit/backend/x86/test/test_jitlog.py
>>>
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>
> 


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/78468041/attachment.html>

From shubharamani at yahoo.com  Fri Dec  9 09:44:06 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 14:44:06 +0000 (UTC)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <360320017.1062071.1481294417004@mail.yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
Message-ID: <266054994.1032093.1481294646674@mail.yahoo.com>

I guess to write this tool, I don't need to actually "dump jit bytecodes" though. I can "dump jit bytecodes" whileI'm writing and debugging my tool. Once I figure out how it works, I just add hooks into the pypy source code base. 

    On Friday, December 9, 2016 6:40 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 I'm building a tool which attempts to map Jit ByteCodes back to the actual function which was called at the Python level.Therefore I need to map the Jit ByteCodes back to the actual function address - the physical address not the virtual.It seems like to build this tool I have to get the rpython version of the open source benchmark - which is not an easy task.I would like to take any old Python application out in the wild and instrument it to dump jitcodes using rpython as an interpreter.?Is this possible ? 

    On Friday, December 9, 2016 6:29 AM, Richard Plangger <planrichi at gmail.com> wrote:
 

 Jitlog is data generated by the JIT compiler. Without a jitdriver, it
will not be able to JIT compile any code, thus you cannot obtain that
file. If you do not have a rpython program, you cannot really generate
the jitlog easily.

I'm unsure what you want to achieve, if you give me more information
what you mean by 'run some open source benchmark' or 'I don't want to
add JItdriver'?

Cheers,
Richard

On 12/09/2016 02:43 PM, Shubha Ramani wrote:
> Is there a way to get Jitlog WITHOUT running it under test and calling metainterp functions ? What if I just want to run some open source benchmark ? What if I don't want to add JItdriver and merge_point to the code and run it "as is" non rpython - can I still dump jitlog ?
> 
> Sent from Shubha Ramani's iPhone 7
> 
>> On Dec 9, 2016, at 1:19 AM, Richard Plangger <planrichi at gmail.com> wrote:
>>
>> Hi,
>>
>> the jit log currently is only functional with the vmprof platform.
>> There was this idea of a command line tool to inspect, but it is not yet
>> finished.
>>
>> Unsure if the test generated file can be uploaded, but you can try:
>>
>> $ pip install vmprof
>> $ python -m jitlog --upload <your-file-file-name>
>>
>> The output of the last program will upload the logfile to vmprof.com and
>> you can inspect it by clicking on the link it prints at the end.
>>
>> Try the following to get a log out of pypy for your program:
>>
>> (pypy-env) $ pip install vmprof
>> (pypy-env) $ pypy -m vmprof --jitlog yourpythonprogram.py
>>
>> Cheers,
>> Richard
>>
>>> On 12/09/2016 02:03 AM, Shubha Ramani via pypy-dev wrote:
>>> I successfully created a file containing binary data, but don't know
>>> what to do with it.
>>> I produced it using
>>> test /opt/pypy/rpython/jit/backend/x86/test/test_jitlog.py
>>>
>>>
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>
> 


   

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/99344eff/attachment-0001.html>

From yury at shurup.com  Fri Dec  9 09:46:13 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 9 Dec 2016 15:46:13 +0100 (CET)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
Message-ID: <alpine.LRH.2.20.1612091544170.1809@vps.zaytsev.net>

On Fri, 9 Dec 2016, Richard Plangger wrote:

> Jitlog is data generated by the JIT compiler. Without a jitdriver, it
> will not be able to JIT compile any code, thus you cannot obtain that
> file. If you do not have a rpython program, you cannot really generate
> the jitlog easily.
>
> I'm unsure what you want to achieve, if you give me more information
> what you mean by 'run some open source benchmark' or 'I don't want to
> add JItdriver'?

I have the impression that he wants

https://vmprof.readthedocs.io/en/latest/jitlog.html

-- 
Sincerely yours,
Yury V. Zaytsev

From shubharamani at yahoo.com  Fri Dec  9 09:51:38 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 14:51:38 +0000 (UTC)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <alpine.LRH.2.20.1612091544170.1809@vps.zaytsev.net>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <alpine.LRH.2.20.1612091544170.1809@vps.zaytsev.net>
Message-ID: <132469053.1020929.1481295098859@mail.yahoo.com>

Hi Yury. You're right that is close to what I want. Just reading that link, it seems that no specialinstrumentation needs to be added to the Python code in order for any old python code out in the wild tobe able to dump JIT compiler logs.?JIT Compiler Logs ? vmprof 0.3 documentation

  
|  
|   |  
JIT Compiler Logs ? vmprof 0.3 documentation
   |  |

  |

 
 

    On Friday, December 9, 2016 6:46 AM, Yury V. Zaytsev <yury at shurup.com> wrote:
 

 On Fri, 9 Dec 2016, Richard Plangger wrote:

> Jitlog is data generated by the JIT compiler. Without a jitdriver, it
> will not be able to JIT compile any code, thus you cannot obtain that
> file. If you do not have a rpython program, you cannot really generate
> the jitlog easily.
>
> I'm unsure what you want to achieve, if you give me more information
> what you mean by 'run some open source benchmark' or 'I don't want to
> add JItdriver'?

I have the impression that he wants

https://vmprof.readthedocs.io/en/latest/jitlog.html

-- 
Sincerely yours,
Yury V. Zaytsev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/5d7feeae/attachment.html>

From yury at shurup.com  Fri Dec  9 09:50:22 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Fri, 9 Dec 2016 15:50:22 +0100 (CET)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <360320017.1062071.1481294417004@mail.yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
Message-ID: <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>

On Fri, 9 Dec 2016, Shubha Ramani via pypy-dev wrote:

> I'm building a tool which attempts to map Jit ByteCodes back to the 
> actual function which was called at the Python level. Therefore I need 
> to map the Jit ByteCodes back to the actual function address - the 
> physical address not the virtual. It seems like to build this tool I 
> have to get the rpython version of the open source benchmark - which is 
> not an easy task. I would like to take any old Python application out in 
> the wild and instrument it to dump jitcodes using rpython as an 
> interpreter. Is this possible ?

It seems to me that you are confused about the relationship between 
RPython and PyPy. The former is the translation toolchain, and the latter 
is the implementation of Python on the basis of this toolchain.

-- 
Sincerely yours,
Yury V. Zaytsev

From shubharamani at yahoo.com  Fri Dec  9 09:55:56 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 14:55:56 +0000 (UTC)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
 <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>
Message-ID: <1125368339.1045474.1481295356042@mail.yahoo.com>

Yury:
I am no longer confused about that. But so far, in order to be able to dump jitcodes, I have to instrument python code with"JitDriver", "merge_point", "meta_interp", etc...under the test directory - and ?the resulting toy python program I write in order to get?the jit dump HAS TO BE rpython compliant. For instance, I tried to write a toy program which calls "divmod" under the test directory -can't do that, since it's disallowed by rpython.
I was responding to Richard's earlier statement:"Jitlog is data generated by the JIT compiler. Without a jitdriver, it
will not be able to JIT compile any code, thus you cannot obtain that
file. If you do not have a rpython program, you cannot really generate
the jitlog easily."
 

    On Friday, December 9, 2016 6:50 AM, Yury V. Zaytsev <yury at shurup.com> wrote:
 

 On Fri, 9 Dec 2016, Shubha Ramani via pypy-dev wrote:

> I'm building a tool which attempts to map Jit ByteCodes back to the 
> actual function which was called at the Python level. Therefore I need 
> to map the Jit ByteCodes back to the actual function address - the 
> physical address not the virtual. It seems like to build this tool I 
> have to get the rpython version of the open source benchmark - which is 
> not an easy task. I would like to take any old Python application out in 
> the wild and instrument it to dump jitcodes using rpython as an 
> interpreter. Is this possible ?

It seems to me that you are confused about the relationship between 
RPython and PyPy. The former is the translation toolchain, and the latter 
is the implementation of Python on the basis of this toolchain.

-- 
Sincerely yours,
Yury V. Zaytsev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/cd9541d2/attachment-0001.html>

From planrichi at gmail.com  Fri Dec  9 11:18:04 2016
From: planrichi at gmail.com (Richard Plangger)
Date: Fri, 9 Dec 2016 17:18:04 +0100
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <132469053.1020929.1481295098859@mail.yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <alpine.LRH.2.20.1612091544170.1809@vps.zaytsev.net>
 <132469053.1020929.1481295098859@mail.yahoo.com>
Message-ID: <c47a2b0b-09f1-97b1-c9aa-b88ea96c9649@gmail.com>

Hi,

you previously wrote:

> I'm building a tool which attempts to map Jit ByteCodes back to the
actual function which was called at the Python level.

jitlog does already do that. If you call your program this way:

(env) $ pip install vmprof
(env) $ pypy -m jitlog --web yourbenchmark.py

you can browse through functions that generated jitcodes, look at the
python bytecode and even assembly.

So there is no need to reinvent the wheel. If you are not happy with
some of the displayed information, I'm happy to review and accept
contributions (or even new features).

Cheers,
Richard

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/3142a133/attachment.sig>

From armin.rigo at gmail.com  Fri Dec  9 11:29:26 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 9 Dec 2016 17:29:26 +0100
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <1125368339.1045474.1481295356042@mail.yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
 <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>
 <1125368339.1045474.1481295356042@mail.yahoo.com>
Message-ID: <CAMSv6X3FG5L8pSKbRqSGpck_rk5afBV5mdob+q=434KTsiFg_w@mail.gmail.com>

Hi,

On 9 December 2016 at 15:55, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:
> I am no longer confused about that.

I think the rest of your mail shows there is still confusion.

> But so far, in order to be able to dump
> jitcodes, I have to instrument python code with
> "JitDriver", "merge_point", "meta_interp", etc...under the test directory -

Maybe you should start again by describing in detail what you want to
do, from the start.

Let me take a wild guess, completely unrelated to the questions you're
asking.  Maybe
your goal is to run a "pypy" binary, which JITs some parts of the user
Python code.
What you want then is to map the raw machine code emitted in memory,
back to Python-
level information.  For example, from an IP, you want an answer like
"this 'add' instruction
comes from JIT-compiling through this Python function, precisely here
[Python bytecode of a CALL], with the call inlined, and running this
other Python
function, precisely here [Python bytecode of a BINARY_ADD].

If that's anywhere close to what you're looking for, then you are
looking at the wrong place.


A bient?t,

Armin.

From shubharamani at yahoo.com  Fri Dec  9 12:34:10 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 09:34:10 -0800
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <CAMSv6X3FG5L8pSKbRqSGpck_rk5afBV5mdob+q=434KTsiFg_w@mail.gmail.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
 <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>
 <1125368339.1045474.1481295356042@mail.yahoo.com>
 <CAMSv6X3FG5L8pSKbRqSGpck_rk5afBV5mdob+q=434KTsiFg_w@mail.gmail.com>
Message-ID: <C5EDA156-A070-4122-82DE-951122F53957@yahoo.com>

Armin you articulated exactly what I want. Please tell me the right place to look then ?

Shubha 

Sent from Shubha Ramani's iPhone 7

> On Dec 9, 2016, at 8:29 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
> 
> Hi,
> 
> On 9 December 2016 at 15:55, Shubha Ramani via pypy-dev
> <pypy-dev at python.org> wrote:
>> I am no longer confused about that.
> 
> I think the rest of your mail shows there is still confusion.
> 
>> But so far, in order to be able to dump
>> jitcodes, I have to instrument python code with
>> "JitDriver", "merge_point", "meta_interp", etc...under the test directory -
> 
> Maybe you should start again by describing in detail what you want to
> do, from the start.
> 
> Let me take a wild guess, completely unrelated to the questions you're
> asking.  Maybe
> your goal is to run a "pypy" binary, which JITs some parts of the user
> Python code.
> What you want then is to map the raw machine code emitted in memory,
> back to Python-
> level information.  For example, from an IP, you want an answer like
> "this 'add' instruction
> comes from JIT-compiling through this Python function, precisely here
> [Python bytecode of a CALL], with the call inlined, and running this
> other Python
> function, precisely here [Python bytecode of a BINARY_ADD].
> 
> If that's anywhere close to what you're looking for, then you are
> looking at the wrong place.
> 
> 
> A bient?t,
> 
> Armin.


From shubharamani at yahoo.com  Fri Dec  9 12:46:45 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 09:46:45 -0800
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <C5EDA156-A070-4122-82DE-951122F53957@yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
 <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>
 <1125368339.1045474.1481295356042@mail.yahoo.com>
 <CAMSv6X3FG5L8pSKbRqSGpck_rk5afBV5mdob+q=434KTsiFg_w@mail.gmail.com>
 <C5EDA156-A070-4122-82DE-951122F53957@yahoo.com>
Message-ID: <4D14253C-946B-46F6-82D1-F2E9C25F3E95@yahoo.com>

I assume it's backend/x86? Why is what I'm doing a waste of time though ?

Sent from Shubha Ramani's iPhone 7

> On Dec 9, 2016, at 9:34 AM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:
> 
> Armin you articulated exactly what I want. Please tell me the right place to look then ?
> 
> Shubha 
> 
> Sent from Shubha Ramani's iPhone 7
> 
>> On Dec 9, 2016, at 8:29 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
>> 
>> Hi,
>> 
>> On 9 December 2016 at 15:55, Shubha Ramani via pypy-dev
>> <pypy-dev at python.org> wrote:
>>> I am no longer confused about that.
>> 
>> I think the rest of your mail shows there is still confusion.
>> 
>>> But so far, in order to be able to dump
>>> jitcodes, I have to instrument python code with
>>> "JitDriver", "merge_point", "meta_interp", etc...under the test directory -
>> 
>> Maybe you should start again by describing in detail what you want to
>> do, from the start.
>> 
>> Let me take a wild guess, completely unrelated to the questions you're
>> asking.  Maybe
>> your goal is to run a "pypy" binary, which JITs some parts of the user
>> Python code.
>> What you want then is to map the raw machine code emitted in memory,
>> back to Python-
>> level information.  For example, from an IP, you want an answer like
>> "this 'add' instruction
>> comes from JIT-compiling through this Python function, precisely here
>> [Python bytecode of a CALL], with the call inlined, and running this
>> other Python
>> function, precisely here [Python bytecode of a BINARY_ADD].
>> 
>> If that's anywhere close to what you're looking for, then you are
>> looking at the wrong place.
>> 
>> 
>> A bient?t,
>> 
>> Armin.
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev


From shubharamani at yahoo.com  Fri Dec  9 16:31:22 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 9 Dec 2016 21:31:22 +0000 (UTC)
Subject: [pypy-dev] how do you parse jit log containing binary dump ?
In-Reply-To: <4D14253C-946B-46F6-82D1-F2E9C25F3E95@yahoo.com>
References: <70460416.752349.1481245421593.ref@mail.yahoo.com>
 <70460416.752349.1481245421593@mail.yahoo.com>
 <9b0f87d4-1215-9cb8-3c41-55aab2c5d382@gmail.com>
 <B2B82837-52C1-473C-904C-94CA1E9E6642@yahoo.com>
 <5caa3b36-d47e-b64b-5ee5-93842fd63a98@gmail.com>
 <360320017.1062071.1481294417004@mail.yahoo.com>
 <alpine.LRH.2.20.1612091547171.1809@vps.zaytsev.net>
 <1125368339.1045474.1481295356042@mail.yahoo.com>
 <CAMSv6X3FG5L8pSKbRqSGpck_rk5afBV5mdob+q=434KTsiFg_w@mail.gmail.com>
 <C5EDA156-A070-4122-82DE-951122F53957@yahoo.com>
 <4D14253C-946B-46F6-82D1-F2E9C25F3E95@yahoo.com>
Message-ID: <810479468.32253.1481319082374@mail.yahoo.com>

I guess I shouldn't be in metainterp at all. But where do I capture OPTIMIZED JIT ? just before it gets converted to ASM but after it's beenoptimized ??In this youtube video:?How the PyPy JIT works? I see the guy pull up "JitViewer" and easily see optimized JIT, assembly and source codeline by line. That is similar to what I want. How do I get it ? I know JitViewer is called vmprof or something now. So yes, Armin, to answer yourconcerns I'd like to take a "call" instruction in the JIT-generated part of ASM and eventually map it back to a Python function call. Am I makingmore sense now ?
  
|  
|   
|   
|   |    |

   |

  |
|  
|    |  
How the PyPy JIT works
 Benjamin Peterson The Python community is abuzz about the major speed gains PyPy can offer pure Python code. But...  |   |

  |

  |

 
 

    On Friday, December 9, 2016 9:46 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 I assume it's backend/x86? Why is what I'm doing a waste of time though ?

Sent from Shubha Ramani's iPhone 7

> On Dec 9, 2016, at 9:34 AM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:
> 
> Armin you articulated exactly what I want. Please tell me the right place to look then ?
> 
> Shubha 
> 
> Sent from Shubha Ramani's iPhone 7
> 
>> On Dec 9, 2016, at 8:29 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
>> 
>> Hi,
>> 
>> On 9 December 2016 at 15:55, Shubha Ramani via pypy-dev
>> <pypy-dev at python.org> wrote:
>>> I am no longer confused about that.
>> 
>> I think the rest of your mail shows there is still confusion.
>> 
>>> But so far, in order to be able to dump
>>> jitcodes, I have to instrument python code with
>>> "JitDriver", "merge_point", "meta_interp", etc...under the test directory -
>> 
>> Maybe you should start again by describing in detail what you want to
>> do, from the start.
>> 
>> Let me take a wild guess, completely unrelated to the questions you're
>> asking.? Maybe
>> your goal is to run a "pypy" binary, which JITs some parts of the user
>> Python code.
>> What you want then is to map the raw machine code emitted in memory,
>> back to Python-
>> level information.? For example, from an IP, you want an answer like
>> "this 'add' instruction
>> comes from JIT-compiling through this Python function, precisely here
>> [Python bytecode of a CALL], with the call inlined, and running this
>> other Python
>> function, precisely here [Python bytecode of a BINARY_ADD].
>> 
>> If that's anywhere close to what you're looking for, then you are
>> looking at the wrong place.
>> 
>> 
>> A bient?t,
>> 
>> Armin.
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161209/623e4ba3/attachment-0001.html>

From shubharamani at yahoo.com  Sun Dec 11 22:32:59 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 12 Dec 2016 03:32:59 +0000 (UTC)
Subject: [pypy-dev] PYPYLOG - get two at once
 jit-backend-dump+jit-backend-addr
References: <1102289739.907971.1481513583929.ref@mail.yahoo.com>
Message-ID: <1102289739.907971.1481513583929@mail.yahoo.com>

I can do??export PYPYLOG=jit-backend-dump:shubha.log and I can do??export PYPYLOG=jit-backend-addr:shubha.logwhile running the attached file which computes Fibonacci numbers. I run the script just by pypy fib.py and I did addthe requisite jitdriver and merge_point stuff.
My question is, is there a way to get both jit-backend-addr and jit-backend-dump in the same log ?
Also what other options are there for PYPYLOG besides the 2 mentioned, jit-backend-dump and jit-backend-addr ?
Thanks,
Shubha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161212/bead4b36/attachment.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fib.py
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161212/bead4b36/attachment.ksh>

From shubharamani at yahoo.com  Sun Dec 11 22:49:03 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 12 Dec 2016 03:49:03 +0000 (UTC)
Subject: [pypy-dev] PYPYLOG - get two at once
 jit-backend-dump+jit-backend-addr
In-Reply-To: <1102289739.907971.1481513583929@mail.yahoo.com>
References: <1102289739.907971.1481513583929.ref@mail.yahoo.com>
 <1102289739.907971.1481513583929@mail.yahoo.com>
Message-ID: <1388938537.921214.1481514543670@mail.yahoo.com>

A search on "debug_start" in the pypy source code base reveals many other log options.So how do you include multiple in one file ?





???

 

    On Sunday, December 11, 2016 7:33 PM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 I can do??export PYPYLOG=jit-backend-dump:shubha.log and I can do??export PYPYLOG=jit-backend-addr:shubha.logwhile running the attached file which computes Fibonacci numbers. I run the script just by pypy fib.py and I did addthe requisite jitdriver and merge_point stuff.
My question is, is there a way to get both jit-backend-addr and jit-backend-dump in the same log ?
Also what other options are there for PYPYLOG besides the 2 mentioned, jit-backend-dump and jit-backend-addr ?
Thanks,
Shubha

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161212/39325c72/attachment.html>

From yury at shurup.com  Mon Dec 12 01:07:27 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Mon, 12 Dec 2016 07:07:27 +0100 (CET)
Subject: [pypy-dev] PYPYLOG - get two at once
 jit-backend-dump+jit-backend-addr
In-Reply-To: <1102289739.907971.1481513583929@mail.yahoo.com>
References: <1102289739.907971.1481513583929.ref@mail.yahoo.com>
 <1102289739.907971.1481513583929@mail.yahoo.com>
Message-ID: <alpine.LRH.2.20.1612120706170.1809@vps.zaytsev.net>

On Mon, 12 Dec 2016, Shubha Ramani via pypy-dev wrote:

> I can do export PYPYLOG=jit-backend-dump:shubha.log and I can do export 
> PYPYLOG=jit-backend-addr:shubha.log while running the attached file 
> which computes Fibonacci numbers. I run the script just by pypy fib.py 
> and I did add the requisite jitdriver and merge_point stuff.

Separate by comma as explained in the documentation that I've linked?

     PYPYLOG=jit-backend-dump:shubha.log,jit-backend-addr:shubha.log

-- 
Sincerely yours,
Yury V. Zaytsev

From armin.rigo at gmail.com  Mon Dec 12 04:10:52 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 12 Dec 2016 10:10:52 +0100
Subject: [pypy-dev] PYPYLOG - get two at once
 jit-backend-dump+jit-backend-addr
In-Reply-To: <alpine.LRH.2.20.1612120706170.1809@vps.zaytsev.net>
References: <1102289739.907971.1481513583929.ref@mail.yahoo.com>
 <1102289739.907971.1481513583929@mail.yahoo.com>
 <alpine.LRH.2.20.1612120706170.1809@vps.zaytsev.net>
Message-ID: <CAMSv6X3VS2nkjixwSfv6=Nxz0FNgyk6Z+=2oOEfd-AYhGJweZA@mail.gmail.com>

Hi,

On 12 December 2016 at 07:07, Yury V. Zaytsev <yury at shurup.com> wrote:
> Separate by comma as explained in the documentation that I've linked?
>
>     PYPYLOG=jit-backend-dump:shubha.log,jit-backend-addr:shubha.log

No, only one file name:

    PYPYLOG=jit-backend-dump,jit-backend-addr:shubha.log

Or use a prefix to enable all related categories:

    PYPYLOG=jit:shubha.log


Armin

From yury at shurup.com  Mon Dec 12 04:23:37 2016
From: yury at shurup.com (Yury V. Zaytsev)
Date: Mon, 12 Dec 2016 10:23:37 +0100 (CET)
Subject: [pypy-dev] PYPYLOG - get two at once
 jit-backend-dump+jit-backend-addr
In-Reply-To: <CAMSv6X3VS2nkjixwSfv6=Nxz0FNgyk6Z+=2oOEfd-AYhGJweZA@mail.gmail.com>
References: <1102289739.907971.1481513583929.ref@mail.yahoo.com>
 <1102289739.907971.1481513583929@mail.yahoo.com>
 <alpine.LRH.2.20.1612120706170.1809@vps.zaytsev.net>
 <CAMSv6X3VS2nkjixwSfv6=Nxz0FNgyk6Z+=2oOEfd-AYhGJweZA@mail.gmail.com>
Message-ID: <alpine.LRH.2.20.1612121022550.1809@vps.zaytsev.net>

On Mon, 12 Dec 2016, Armin Rigo wrote:

> Hi,
>
> On 12 December 2016 at 07:07, Yury V. Zaytsev <yury at shurup.com> wrote:
>> Separate by comma as explained in the documentation that I've linked?
>>
>>     PYPYLOG=jit-backend-dump:shubha.log,jit-backend-addr:shubha.log
>
> No, only one file name:
>
>    PYPYLOG=jit-backend-dump,jit-backend-addr:shubha.log

Sorry, copy & paste error!

> Or use a prefix to enable all related categories:
>
>    PYPYLOG=jit:shubha.log

Nice, didn't know about that one, thanks :-)

-- 
Sincerely yours,
Yury V. Zaytsev

From shubharamani at yahoo.com  Mon Dec 12 07:17:54 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 12 Dec 2016 04:17:54 -0800
Subject: [pypy-dev] Where should I look then !
Message-ID: <00D2157B-5E03-41C2-B4EA-D70A7311F650@yahoo.com>

To do this: 


> On Dec 9, 2016, at 9:34 AM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:
> 
> Armin you articulated exactly what I want. Please tell me the right place to look then ?
> 
> Shubha 
> 
> Sent from Shubha Ramani's iPhone 7
> 
>> On Dec 9, 2016, at 8:29 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
>> 
>> Hi,
>> 
>> On 9 December 2016 at 15:55, Shubha Ramani via pypy-dev
>> <pypy-dev at python.org> wrote:
>>> I am no longer confused about that.
>> 
>> I think the rest of your mail shows there is still confusion.
>> 
>>> But so far, in order to be able to dump
>>> jitcodes, I have to instrument python code with
>>> "JitDriver", "merge_point", "meta_interp", etc...under the test directory -
>> 
>> Maybe you should start again by describing in detail what you want to
>> do, from the start.
>> 
>> Let me take a wild guess, completely unrelated to the questions you're
>> asking.  Maybe
>> your goal is to run a "pypy" binary, which JITs some parts of the user
>> Python code.
>> What you want then is to map the raw machine code emitted in memory,
>> back to Python-
>> level information.  For example, from an IP, you want an answer like
>> "this 'add' instruction
>> comes from JIT-compiling through this Python function, precisely here
>> [Python bytecode of a CALL], with the call inlined, and running this
>> other Python
>> function, precisely here [Python bytecode of a BINARY_ADD].
>> 
>> If that's anywhere close to what you're looking for, then you are
>> looking at the wrong place.
>> 
>> 
>> A bient?t,
>> 
>> Armin.
> 

Sent from Shubha Ramani's iPhone 7
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161212/4e7b0603/attachment.html>

From armin.rigo at gmail.com  Mon Dec 12 14:10:59 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 12 Dec 2016 20:10:59 +0100
Subject: [pypy-dev] Where should I look then !
In-Reply-To: <00D2157B-5E03-41C2-B4EA-D70A7311F650@yahoo.com>
References: <00D2157B-5E03-41C2-B4EA-D70A7311F650@yahoo.com>
Message-ID: <CAMSv6X1ZSLHn2DSku+_DcQCSRgUc9RSGe9X8psyBz8Ms4sxd1w@mail.gmail.com>

Hi Shubha,

Sorry, you are asking many questions and I think you are not carefully
reading the answers.  Similarly, you are insisting that you've read
*all* the docs---which would be good, except that at the same time
some of yours questions have clear answers in the docs.

About your precise question: you should re-read some past answers,
e.g. Richard's December 9th, and then look in the source code how the
vmprof integration is done through RPython.  You might also be
interested in the pypy-dev thread from August about VTune, which
contains the answers to similar questions asked by other Intel people
at the time.

In the source code, all relevant places can be found from the
high-level pypy.module._vmprof, to the RPython-level
rpython.rlib.rvmprof; and x86.assembler._call_header_vmprof() which
generates in the assembler frames some structure with
VMPROF_JITTED_TAG, which is read by logic in rpython.rlib.rvmprof.  In
general, grepping for "vmprof" or "rvmprof" throughout the source code
of RPython is a good start.

I will now assume that you have read the answer above and actually
seriously dig into the various parts I mention, before asking more
questions.  Of course, you can keep asking anyway, but you need to
accept that we may not have the time and energy to keep answering you.
Asking a deluge of questions and then pressuring to get prompt
answers, that is the perfect way to get ignored in the open source
world---nobody *has* to answer you.

Also, I have nothing against helping someone building VTune support
inside PyPy, but if you read the August thread I mentioned above you'd
understand that I see VTune in a negative light right now: it pretends
to be more open-source-friendly that it is.  As far as I am concerned,
I am helping you build support specifically for an Intel commercial
product, and I am doing that without getting paid.  I am fine with
doing that to some extend, but it is an unstable situation.


A bient?t,

Armin.

From shubharamani at yahoo.com  Mon Dec 12 14:44:03 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Mon, 12 Dec 2016 19:44:03 +0000 (UTC)
Subject: [pypy-dev] Where should I look then !
In-Reply-To: <CAMSv6X1ZSLHn2DSku+_DcQCSRgUc9RSGe9X8psyBz8Ms4sxd1w@mail.gmail.com>
References: <00D2157B-5E03-41C2-B4EA-D70A7311F650@yahoo.com>
 <CAMSv6X1ZSLHn2DSku+_DcQCSRgUc9RSGe9X8psyBz8Ms4sxd1w@mail.gmail.com>
Message-ID: <715470829.1303241.1481571843937@mail.yahoo.com>

Armin I don't blame you for being upset with me. I'm sorry...I will lay low and dive deep into the pointers you've just nowmentioned in this email. I'm sorry, I didn't know about the August Vtune thread.Please accept my apologies.
Shubha 

    On Monday, December 12, 2016 11:11 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
 

 Hi Shubha,

Sorry, you are asking many questions and I think you are not carefully
reading the answers.? Similarly, you are insisting that you've read
*all* the docs---which would be good, except that at the same time
some of yours questions have clear answers in the docs.

About your precise question: you should re-read some past answers,
e.g. Richard's December 9th, and then look in the source code how the
vmprof integration is done through RPython.? You might also be
interested in the pypy-dev thread from August about VTune, which
contains the answers to similar questions asked by other Intel people
at the time.

In the source code, all relevant places can be found from the
high-level pypy.module._vmprof, to the RPython-level
rpython.rlib.rvmprof; and x86.assembler._call_header_vmprof() which
generates in the assembler frames some structure with
VMPROF_JITTED_TAG, which is read by logic in rpython.rlib.rvmprof.? In
general, grepping for "vmprof" or "rvmprof" throughout the source code
of RPython is a good start.

I will now assume that you have read the answer above and actually
seriously dig into the various parts I mention, before asking more
questions.? Of course, you can keep asking anyway, but you need to
accept that we may not have the time and energy to keep answering you.
Asking a deluge of questions and then pressuring to get prompt
answers, that is the perfect way to get ignored in the open source
world---nobody *has* to answer you.

Also, I have nothing against helping someone building VTune support
inside PyPy, but if you read the August thread I mentioned above you'd
understand that I see VTune in a negative light right now: it pretends
to be more open-source-friendly that it is.? As far as I am concerned,
I am helping you build support specifically for an Intel commercial
product, and I am doing that without getting paid.? I am fine with
doing that to some extend, but it is an unstable situation.


A bient?t,

Armin.

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161212/b28cd224/attachment.html>

From yzhang2 at corp.netease.com  Tue Dec 13 23:19:14 2016
From: yzhang2 at corp.netease.com (=?GBK?B?1cXSqw==?=)
Date: Wed, 14 Dec 2016 12:19:14 +0800 (GMT+08:00)
Subject: [pypy-dev] PYPY:why withprebuiltint option is off default? The
 speed is slower after I open it.
Message-ID: <64f8a7c5.6de3.158fb8e432c.Coremail.yzhang2@corp.netease.com>

Hi, every experts:
   Recently I'm trying to move my old project from python to pypy.  In the past, I have do some optimizations for python interpreter. One Important thing is modify python default integer object cache range(default is from -5 to 257),  my config is  -100 to 1024 * 1024,  this range covers most of the my app logics. The benchmark shows that it saves a lot of memory ,  and speedup the program(mainly benefits from avoiding large amount of  creating intobject and releasing intobject) .

  BUT, in pypy, I found the withprebuiltint option is set off default. While I tried open this option and add translateoption : --prebuiltintfrom -5  --prebuiltintfrom 1048576, the benchmark shows that it saves memory(OK, it's expected), BUT the speed is SLOWER. 
  
  I check the pypy sourcecode, but I have not find out the problem so far. So, my doubt is: why withprebuiltint option is off default?  Maybe the performance is a problem as known? 

#############################################################################################
#test_speed4_prebuit_int.py  
def foo():
        j = 0 
        for i in xrange(1024 * 1024):
                j = i -1
                j = i + 1    

import timeit
print timeit.repeat(foo, repeat = 3, number= 100)

user:~/pypy_run$ official-pypy/bin/pypy-c -B test_speed4_prebuit_int.py  #default pypy
[0.21457695960998535, 0.21348810195922852, 0.21274185180664062]

user:~/pypy_run$ pypy-build-by-pypy-test/bin/pypy-c -B   test_speed4_prebuit_int.py  #--prebuiltintfrom -5  --prebuiltintfrom 1048576
[0.7308619022369385, 0.7308559417724609, 0.7256178855895996]
#almost 3 times slower


#############################################################################################
#test_mem4_prebuilt_int.py
listlen = 1024
itemlen = 1024 * 100 
x = 1 
a = [None] * listlen

def init_data():
        for i in xrange(listlen):
                a[i] = [None] * itemlen
def create_data():
        for i in xrange(listlen):
                for j in xrange(itemlen - 1):
                        a[i][j] = x + 0 

init_data()
create_data()
###mem test is expected, no problem



Thanks regards!

shadow
2016-12-14






From armin.rigo at gmail.com  Wed Dec 14 07:14:48 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Wed, 14 Dec 2016 13:14:48 +0100
Subject: [pypy-dev] PYPY:why withprebuiltint option is off default? The
 speed is slower after I open it.
In-Reply-To: <64f8a7c5.6de3.158fb8e432c.Coremail.yzhang2@corp.netease.com>
References: <64f8a7c5.6de3.158fb8e432c.Coremail.yzhang2@corp.netease.com>
Message-ID: <CAMSv6X1hV2jinNa2yxFhFjUxCU62pMEgdbkarVWPm+SMVs=sHA@mail.gmail.com>

Hi,

On 14 December 2016 at 05:19, ?? <yzhang2 at corp.netease.com> wrote:
>   BUT, in pypy, I found the withprebuiltint option is set off default. While I tried open this option and add translateoption : --prebuiltintfrom -5  --prebuiltintfrom 1048576, the benchmark shows that it saves memory(OK, it's expected), BUT the speed is SLOWER.

The topic of this message says it all:

> why withprebuiltint option is off default? The speed is slower after I open it.

That's why it is off by default: because it is slower.  PyPy differs
from CPython in two important aspects: it has a GC that is well-tuned
to creating and freeing objects at a high frequency; and it has a JIT
compiler that is able to remove allocations.  Each of these points
alone makes this optimization worthless (and others too).

In your trivial benchmark, all the integer objects are removed by the
JIT compiler, because it notices that they are created and then
forgotten very soon.  The machine code "explodes" the objects into a
single register each, which contains the integer value; this gives
basically C-like performance.  By contrast, if you enable
--withprebuiltint, then the code is much slower: it needs to add
checks that the numbers are within the specified range, and if they
are it needs to load the prebuilt objects' addresses and manipulate
them.  That's nonsense for JIT-generated machine code, and that's why
it is disabled by default.  It is only useful if you want to
absolutely minimize memory usage (in which case you make a slow pypy
with no JIT at all, with ``-Omem``).

In theory we could experiment with enabling ``--withprebuiltint`` only
when the JIT is not running.  But I strongly suspect that it is not
faster even without the JIT---our GC is good at that job.  Try to run
your benchmark with ``pypy --jit off``?


A bient?t,

Armin.

From william.leslie.ttg at gmail.com  Wed Dec 14 07:24:22 2016
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Wed, 14 Dec 2016 23:24:22 +1100
Subject: [pypy-dev] PYPY:why withprebuiltint option is off default? The
 speed is slower after I open it.
In-Reply-To: <64f8a7c5.6de3.158fb8e432c.Coremail.yzhang2@corp.netease.com>
References: <64f8a7c5.6de3.158fb8e432c.Coremail.yzhang2@corp.netease.com>
Message-ID: <CAHgd1hHowVn39Nh4c0PxRkcu+bBsn1D1PGsSCLfipueg4WS6xw@mail.gmail.com>

On 14 December 2016 at 15:19, ?? <yzhang2 at corp.netease.com> wrote:
> Hi, every experts:
>    Recently I'm trying to move my old project from python to pypy.  In the past, I have do some optimizations for python interpreter. One Important thing is modify python default integer object cache range(default is from -5 to 257),  my config is  -100 to 1024 * 1024,  this range covers most of the my app logics. The benchmark shows that it saves a lot of memory ,  and speedup the program(mainly benefits from avoiding large amount of  creating intobject and releasing intobject) .
>
>   BUT, in pypy, I found the withprebuiltint option is set off default. While I tried open this option and add translateoption : --prebuiltintfrom -5  --prebuiltintfrom 1048576, the benchmark shows that it saves memory(OK, it's expected), BUT the speed is SLOWER.
>

Allocating a new object in PyPy is faster to start with, especially
with the default GC, and because of the GC releasing short-lived
intobject costs nothing.  Additionally, there are lots of places where
ints do not need an object.  PyPy's specialised dictionary for laying
out class instances (mapdict) leaves only a platform integer in many
cases.  Similarly for lists, lists of just integers have a special
compact representation in a default PyPy build.  Hot loops also get
compiled, and any integers that don't escape the loop are unboxed and
need to object to represent them.

Prebuilt ints also have another cost that works against the layouts
and compiled code above:  everywhere an operation could produce a
prebuilt intobject, an extra conditional check has to be performed.
Having prebuilt integers turned off removes this condition from warm
code.

>   I check the pypy sourcecode, but I have not find out the problem so far. So, my doubt is: why withprebuiltint option is off default?  Maybe the performance is a problem as known?
>

Yes.  Prebuilt ints are only really an optimisation on CPython, PyPy
has better optimisations that work well with GC.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.

From armin.rigo at gmail.com  Thu Dec 15 08:28:10 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Thu, 15 Dec 2016 14:28:10 +0100
Subject: [pypy-dev] cling-support
Message-ID: <CAMSv6X38xAZdd8+J9DyMrdZjTCwubf1z1D_4KQjebLuBfvKM1w@mail.gmail.com>

Hi Wim,

Great that cling-support is merged!  Could you (1) document the branch
inside pypy/doc/whatsnew-head.rst, and (2) look at the failure on
32-bit?  It seems that running py.test module/cppyy crashes with exit
code 3, according to
http://buildbot.pypy.org/summary/longrepr?testname=&builder=own-linux-x86-32&build=4857&mod=module%2Fcppyy%2Ftest
.  Thanks!


A bient?t,

Armin.

From wlavrijsen at lbl.gov  Thu Dec 15 09:59:21 2016
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 15 Dec 2016 06:59:21 -0800 (PST)
Subject: [pypy-dev] cling-support
In-Reply-To: <CAMSv6X38xAZdd8+J9DyMrdZjTCwubf1z1D_4KQjebLuBfvKM1w@mail.gmail.com>
References: <CAMSv6X38xAZdd8+J9DyMrdZjTCwubf1z1D_4KQjebLuBfvKM1w@mail.gmail.com>
Message-ID: <alpine.LSU.2.11.1612150652200.15854@cp3-jyni>

Hi Armin,

> Could you (1) document the branch inside pypy/doc/whatsnew-head.rst

yes; I'm still working on documentation and the pypi package upload. Almost
there ...

, and (2) look at the failure on
> 32-bit?  It seems that running py.test module/cppyy crashes with exit
> code 3, according to
> http://buildbot.pypy.org/summary/longrepr?testname=&builder=own-linux-x86-32&build=4857&mod=module%2Fcppyy%2Ftest

The error is:

  [platform:Error] cc1plus: error: unrecognized command line option '-std=c++11'

How old is that gcc? I can probably do without c++11 for the dummy/test
backend, but not for the actual backend (in the pip package).

Thanks,
      Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From armin.rigo at gmail.com  Thu Dec 15 10:49:17 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Thu, 15 Dec 2016 16:49:17 +0100
Subject: [pypy-dev] cling-support
In-Reply-To: <alpine.LSU.2.11.1612150652200.15854@cp3-jyni>
References: <CAMSv6X38xAZdd8+J9DyMrdZjTCwubf1z1D_4KQjebLuBfvKM1w@mail.gmail.com>
 <alpine.LSU.2.11.1612150652200.15854@cp3-jyni>
Message-ID: <CAMSv6X2umka000ge3ZGg4hZoY3MMqcr+gS+qANkdrjBxnjQ2dw@mail.gmail.com>

Hi Wim,

On 15 December 2016 at 15:59,  <wlavrijsen at lbl.gov> wrote:
> The error is:
>
>  [platform:Error] cc1plus: error: unrecognized command line option
> '-std=c++11'
>
> How old is that gcc? I can probably do without c++11 for the dummy/test
> backend, but not for the actual backend (in the pip package).

Ah, that's an old gcc indeed.  Maybe detect this case and skip all tests?


A bient?t,

Armin.

From frankw at mit.edu  Thu Dec 15 15:06:35 2016
From: frankw at mit.edu (Frank Wang)
Date: Thu, 15 Dec 2016 15:06:35 -0500
Subject: [pypy-dev] Adding init/variables to W_Root
Message-ID: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>

Hi,

I don't know if this is the proper list to ask this question.

I am trying to add a dictionary to W_Root. The reason for this is that I
want all interpreter objects to contain this dictionary. This is for a
research project to implement some form of information flow control in PyPy.

Right now, I know W_Root has no __init__ function, and when I try to add
one. I run into all sorts of problems because TypeDef calls W_Root.__new__
with some parameters, but "new" function seems to be called nowhere else.

Any thoughts on this would be helpful.

Thanks,
Frank
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161215/6ad7822b/attachment.html>

From wlavrijsen at lbl.gov  Thu Dec 15 17:15:37 2016
From: wlavrijsen at lbl.gov (wlavrijsen at lbl.gov)
Date: Thu, 15 Dec 2016 14:15:37 -0800 (PST)
Subject: [pypy-dev] cling-support
In-Reply-To: <CAMSv6X2umka000ge3ZGg4hZoY3MMqcr+gS+qANkdrjBxnjQ2dw@mail.gmail.com>
References: <CAMSv6X38xAZdd8+J9DyMrdZjTCwubf1z1D_4KQjebLuBfvKM1w@mail.gmail.com>
 <alpine.LSU.2.11.1612150652200.15854@cp3-jyni>
 <CAMSv6X2umka000ge3ZGg4hZoY3MMqcr+gS+qANkdrjBxnjQ2dw@mail.gmail.com>
Message-ID: <alpine.LSU.2.11.1612151408180.3945@cp3-jyni>

Hi Armin,

good idea; done. Documentation and release notes are also updated (mostly
removal of caveats that no longer are :) ) and the PyPI package has also
been uploaded (PyPy-cppyy-backend).

Now that I know how to do PyPI packages (sortof, anyway), I should find the
time to do the CPython one as well ...

That and a CFFI-style interface on top of cppyy. Wouldn't be as light-weight
but that can always be fixed in due time.

Aside, I got to wave the PyPy flag for a bit at PyHPC:
    http://conferences.computer.org/pyhpc/2016/papers/5220a027.pdf

Surprising how few folks had even heard of PyPy. :(

Best regards,
            Wim
-- 
WLavrijsen at lbl.gov    --    +1 (510) 486 6411    --    www.lavrijsen.net

From armin.rigo at gmail.com  Fri Dec 16 11:19:17 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 16 Dec 2016 17:19:17 +0100
Subject: [pypy-dev] Adding init/variables to W_Root
In-Reply-To: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
References: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
Message-ID: <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>

Hi Frank,

On 15 December 2016 at 21:06, Frank Wang <frankw at mit.edu> wrote:
> Right now, I know W_Root has no __init__ function, and when I try to add
> one. I run into all sorts of problems because TypeDef calls W_Root.__new__
> with some parameters, but "new" function seems to be called nowhere else.

I suspect there is no clean way to add a __init__() method to W_Root.
You could use default attributes instead:

class W_Root:
    _my_extra_dict = None

    def get_extra_dict(self):
        if self._my_extra_dict is None:
            self._my_extra_dict = {}
        return self._my_extra_dict



A bient?t,

Armin.

From frankw at mit.edu  Fri Dec 16 12:05:56 2016
From: frankw at mit.edu (Frank Wang)
Date: Fri, 16 Dec 2016 12:05:56 -0500
Subject: [pypy-dev] Adding init/variables to W_Root
In-Reply-To: <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>
References: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
 <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>
Message-ID: <CAD=7XjN9XjAMYvotxU_6fZ6=D4+0y67RmZLDg=_tE-TkimSsZA@mail.gmail.com>

Hi Armin,

Thanks for the suggestion! I'll see if that works. Just to make sure. This
will give me an instance variable? I need values of the extra dict to be
different for different instantiations of W_Root.

Frank

On Fri, Dec 16, 2016 at 11:19 AM, Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi Frank,
>
> On 15 December 2016 at 21:06, Frank Wang <frankw at mit.edu> wrote:
> > Right now, I know W_Root has no __init__ function, and when I try to add
> > one. I run into all sorts of problems because TypeDef calls
> W_Root.__new__
> > with some parameters, but "new" function seems to be called nowhere else.
>
> I suspect there is no clean way to add a __init__() method to W_Root.
> You could use default attributes instead:
>
> class W_Root:
>     _my_extra_dict = None
>
>     def get_extra_dict(self):
>         if self._my_extra_dict is None:
>             self._my_extra_dict = {}
>         return self._my_extra_dict
>
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161216/c8a527b2/attachment.html>

From cfbolz at gmx.de  Fri Dec 16 17:20:34 2016
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Fri, 16 Dec 2016 23:20:34 +0100
Subject: [pypy-dev] Adding init/variables to W_Root
In-Reply-To: <CAD=7XjN9XjAMYvotxU_6fZ6=D4+0y67RmZLDg=_tE-TkimSsZA@mail.gmail.com>
References: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
 <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>
 <CAD=7XjN9XjAMYvotxU_6fZ6=D4+0y67RmZLDg=_tE-TkimSsZA@mail.gmail.com>
Message-ID: <d426e74a-3d26-4bcb-98d1-dd6e797b07f3@email.android.com>

Hi Frank, 

Yes, you'll get an instance variable that way. 

Cheers, 
Carl Friedrich

On December 16, 2016 6:05:56 PM GMT+01:00, Frank Wang <frankw at mit.edu> wrote:
>Hi Armin,
>
>Thanks for the suggestion! I'll see if that works. Just to make sure.
>This
>will give me an instance variable? I need values of the extra dict to
>be
>different for different instantiations of W_Root.
>
>Frank
>
>On Fri, Dec 16, 2016 at 11:19 AM, Armin Rigo <armin.rigo at gmail.com>
>wrote:
>
>> Hi Frank,
>>
>> On 15 December 2016 at 21:06, Frank Wang <frankw at mit.edu> wrote:
>> > Right now, I know W_Root has no __init__ function, and when I try
>to add
>> > one. I run into all sorts of problems because TypeDef calls
>> W_Root.__new__
>> > with some parameters, but "new" function seems to be called nowhere
>else.
>>
>> I suspect there is no clean way to add a __init__() method to W_Root.
>> You could use default attributes instead:
>>
>> class W_Root:
>>     _my_extra_dict = None
>>
>>     def get_extra_dict(self):
>>         if self._my_extra_dict is None:
>>             self._my_extra_dict = {}
>>         return self._my_extra_dict
>>
>>
>>
>> A bient?t,
>>
>> Armin.
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161216/5997aa3b/attachment.html>

From hgualandi at inf.puc-rio.br  Fri Dec 16 17:41:10 2016
From: hgualandi at inf.puc-rio.br (hgualandi at inf.puc-rio.br)
Date: Fri, 16 Dec 2016 20:41:10 -0200 (BRST)
Subject: [pypy-dev] Why does pypy include my code three times in the
 JIT-capable executable?
Message-ID: <53782.139.82.100.2.1481928070.squirrel@webmail.inf.puc-rio.br>

Hi there,

I have a question about pypy's compilation process when producing an
executable from an RPython source program.

Apparently when I compile my program with --opt=JIT, my code gets included
3 times in the resulting executable. I made a little experiment where I
included a magic number in my RPython code and counted how often it showed
up in the generated executable.

With ahead of time compilation (--opt=3) my magic number appeared once:

    $ PERLIO=:raw perl -nE '$c++ while m/\xef\xbe\xad\xde/g; END{say $c}'
./rpython-deadbeef.aot
    1

And with --opt=JIT it appeared 3 times:

    $ PERLIO=:raw perl -nE '$c++ while m/\xef\xbe\xad\xde/g; END{say $c}'
./rpython-deadbeef.jit
    3

When I investigated, it turns out that the JIT-capable executable has one
copy of my program in bytecode form and two copies in machine code form.
To understand the machine-code copies I inspected the intermediate
"implement.c" file that pypy generated and found out that my interpreter
code appears in two functions, one named pypy_g_mainloop_1 and one named
pypy_g_portal. These two functions were completely identical, other than
the function names that show up in the debugging messages.

I understand why pypy would create a separate bytecode version of my
program but why is it generating two seemingly-identical machine code
versions of it?


Thanks in advance,
Hugo

From armin.rigo at gmail.com  Sat Dec 17 05:59:10 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 17 Dec 2016 11:59:10 +0100
Subject: [pypy-dev] Dontbug: A reversible debugger for PHP (similar in
 concept to RevDB for Python/PyPy)
In-Reply-To: <CAHZjNtZQhJVjooKMOdyE53z9REdpf6NPBrjp7R9AUwiVCJ=duQ@mail.gmail.com>
References: <CAHZjNtbgPcUX0R7D8Ws1x6e0obDRbEBHrFAz4-cjGujbAkApAQ@mail.gmail.com>
 <CAK5idxQfgRjTiD+QZBGuqu8ucREY=me7_pN=nDxy2jf=uQop-Q@mail.gmail.com>
 <CAHZjNtZQhJVjooKMOdyE53z9REdpf6NPBrjp7R9AUwiVCJ=duQ@mail.gmail.com>
Message-ID: <CAMSv6X1JvdZk7R9P8XzKZ_61JO5BXaq7JK-xojye468GSNYeSA@mail.gmail.com>

Hi Sidharth,

Sorry for answering you only now.

Dontbug is interesting.  It offers roughly the same set of basic
features than RevDB or UndoDB or rr itself.  It is interesting indeed
that reverse debugging is slowly becoming more mainstream, and
standardizing a bit the supported features.  For example, when using
RevDB I found watchpoints to be one of the most useful features.  I
see that Dontbug doesn't support them yet, but plans to.  Great :-)

Implementation-wise, Dontbug is a very different approach than
RevDB.  RevDB is much easier to port than rr: for example it works on
OS/X too, and could be made to work on Windows with more efforts.  It
doesn't require particular OS support, apart from being based for now
on fork().  In the typical PyPy way, it works by systematically
transforming some constructs in the RPython source code of PyPy; this
is something that you can't really do if you start with C code.

On the other hand, with Dontbug I imagine you built a reverse debugger
for PHP by starting with C source code of the PHP implementation,
building on top of what rr provides.  There is a chance that someone
will also do with CPython some day, as well as other interpreters.

Maybe we should try to push RevDB more, but right now it seems not to
be used or known a lot.


On 21 October 2016 at 08:36, Sidharth Kshatriya <sid.kshatriya at gmail.com> wrote:
> I also noticed some references to UndoDB usage on the PyPy project. How has
> your team's experience been with UndoDB+PyPy in general? It would
> interesting to learn about your experiences there...

Yes, we use UndoDB very occasionally to debug hard issues inside the
generated C code of PyPy itself (which is unrelated to RevDB, whose
goal is to be used by Python users).  UndoDB is rather stable
nowadays.  I also tried rr for that once or twice.  But needing that
is rare enough that I'm at most an occasional user of both.


A bient?t,

Armin.

From shubharamani at yahoo.com  Sat Dec 17 14:32:01 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Sat, 17 Dec 2016 19:32:01 +0000 (UTC)
Subject: [pypy-dev] which API returns Python stack in codemap.c given pc ?
References: <1687153522.872443.1482003121391.ref@mail.yahoo.com>
Message-ID: <1687153522.872443.1482003121391@mail.yahoo.com>

I have studied?rpython/jit/backend/llsupport/src/codemap.c.I am nt finding the API which returns the Python stack in codemap.c.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161217/48a87f3f/attachment.html>

From shubharamani at yahoo.com  Sat Dec 17 18:51:53 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Sat, 17 Dec 2016 23:51:53 +0000 (UTC)
Subject: [pypy-dev] no module named 'backports' error when vmprof executed
 with pypy
References: <531662631.966991.1482018713768.ref@mail.yahoo.com>
Message-ID: <531662631.966991.1482018713768@mail.yahoo.com>

shubha at shubha-Z170X-UD5:/opt/vtune_pypy$ pypy-v -m vmprof fib.py/usr/bin/pypy-v: No module named backports; 'vmprof' is a package and cannot be directly executed
However when I do python -m vmprof fib.py it works.?
Based on a quick Google search there doesn't seem to be a package called 'backports' which caneasily be installed via pip or easy_install.
I'm on Ubuntu 16.04. Any clues ?
Thanks,
Shubha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161217/6c4c32dc/attachment.html>

From shubharamani at yahoo.com  Sat Dec 17 18:54:45 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Sat, 17 Dec 2016 23:54:45 +0000 (UTC)
Subject: [pypy-dev] Fw: no module named 'backports' error when vmprof
 executed with pypy
In-Reply-To: <531662631.966991.1482018713768@mail.yahoo.com>
References: <531662631.966991.1482018713768.ref@mail.yahoo.com>
 <531662631.966991.1482018713768@mail.yahoo.com>
Message-ID: <449807169.970964.1482018885369@mail.yahoo.com>

Never mind.?
Easily solved by:pip install backports.ssl-match-hostname
AND?pip install backports.shutil_which


     On Saturday, December 17, 2016 3:52 PM, Shubha Ramani via pypy-dev <pypy-dev at python.org> wrote:
 

 shubha at shubha-Z170X-UD5:/opt/vtune_pypy$ pypy-v -m vmprof fib.py/usr/bin/pypy-v: No module named backports; 'vmprof' is a package and cannot be directly executed
However when I do python -m vmprof fib.py it works.?
Based on a quick Google search there doesn't seem to be a package called 'backports' which caneasily be installed via pip or easy_install.
I'm on Ubuntu 16.04. Any clues ?
Thanks,
Shubha
_______________________________________________
pypy-dev mailing list
pypy-dev at python.org
https://mail.python.org/mailman/listinfo/pypy-dev


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161217/72c7c67c/attachment.html>

From armin.rigo at gmail.com  Sun Dec 18 05:33:26 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 18 Dec 2016 11:33:26 +0100
Subject: [pypy-dev] which API returns Python stack in codemap.c given pc
 ?
In-Reply-To: <1687153522.872443.1482003121391@mail.yahoo.com>
References: <1687153522.872443.1482003121391.ref@mail.yahoo.com>
 <1687153522.872443.1482003121391@mail.yahoo.com>
Message-ID: <CAMSv6X1V-tBubXo8UpujHzqK7ia8=eSUVQ13Sw+Wq7Z8e31Y4A@mail.gmail.com>

Hi Shubha,

On 17 December 2016 at 20:32, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:
> I have studied rpython/jit/backend/llsupport/src/codemap.c.
> I am nt finding the API which returns the Python stack in codemap.c.

There is no such knowledge as the "Python stack" inside the JIT.  Keep
in mind that the JIT is generic.  You have to study the (generic)
rpython.rlib.rvmprof and the (Python-specific) pypy.module._vmprof to
find out how together they reconstruct this information.


A bient?t,

Armin.

From armin.rigo at gmail.com  Sun Dec 18 05:39:25 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sun, 18 Dec 2016 11:39:25 +0100
Subject: [pypy-dev] Why does pypy include my code three times in the
 JIT-capable executable?
In-Reply-To: <53782.139.82.100.2.1481928070.squirrel@webmail.inf.puc-rio.br>
References: <53782.139.82.100.2.1481928070.squirrel@webmail.inf.puc-rio.br>
Message-ID: <CAMSv6X27xJL9e62Lz19qvu8n8xiT12c+9y9B4ece-uFXQiR1LA@mail.gmail.com>

Hi,

On 16 December 2016 at 23:41,  <hgualandi at inf.puc-rio.br> wrote:
> Apparently when I compile my program with --opt=JIT, my code gets included
> 3 times in the resulting executable.

Likely, you are seeing two C copies of the function because of the way
the JIT is built.  You may get two very similar copies, but only of
the function that contains jitdriver.jit_merge_point().  You only get
one copy of all the rest.

The reason is that there is the normal function, and another
reorganized copy which can be called in some cases, like when
execution needs to exit the JIT and jump back to the interpreter.
This copy starts from the jit_merge_point instead of starting from the
normal entry point of the function.  In some cases the two versions
can be very similar, but for example, if the function does things
before entering the loop, then these things will not be found in the
JIT-generated copy.


A bient?t,

Armin.

From shubharamani at yahoo.com  Sun Dec 18 08:47:59 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Sun, 18 Dec 2016 05:47:59 -0800
Subject: [pypy-dev] which API returns Python stack in codemap.c given pc
 ?
In-Reply-To: <CAMSv6X1V-tBubXo8UpujHzqK7ia8=eSUVQ13Sw+Wq7Z8e31Y4A@mail.gmail.com>
References: <1687153522.872443.1482003121391.ref@mail.yahoo.com>
 <1687153522.872443.1482003121391@mail.yahoo.com>
 <CAMSv6X1V-tBubXo8UpujHzqK7ia8=eSUVQ13Sw+Wq7Z8e31Y4A@mail.gmail.com>
Message-ID: <356395B0-0FE7-4C62-8C0F-C09375D82B15@yahoo.com>

Hi Armin. Yes I've studied the codes you mention deeply. They are fairly straightforward to follow so thank you for the tips.

But I'm merely responding to a statement you made in the August 2016 thread. See below. 

https://mail.python.org/pipermail/pypy-dev/2016-August/014597.html

Thanks Shubharamani 

> On Dec 18, 2016, at 2:33 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
> 
> Hi Shubha,
> 
> On 17 December 2016 at 20:32, Shubha Ramani via pypy-dev
> <pypy-dev at python.org> wrote:
>> I have studied rpython/jit/backend/llsupport/src/codemap.c.
>> I am nt finding the API which returns the Python stack in codemap.c.
> 
> There is no such knowledge as the "Python stack" inside the JIT.  Keep
> in mind that the JIT is generic.  You have to study the (generic)
> rpython.rlib.rvmprof and the (Python-specific) pypy.module._vmprof to
> find out how together they reconstruct this information.
> 
> 
> A bient?t,
> 
> Armin.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161218/c73dd531/attachment.html>

From frankw at mit.edu  Sun Dec 18 23:29:30 2016
From: frankw at mit.edu (Frank Wang)
Date: Sun, 18 Dec 2016 20:29:30 -0800
Subject: [pypy-dev] Adding init/variables to W_Root
In-Reply-To: <d426e74a-3d26-4bcb-98d1-dd6e797b07f3@email.android.com>
References: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
 <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>
 <CAD=7XjN9XjAMYvotxU_6fZ6=D4+0y67RmZLDg=_tE-TkimSsZA@mail.gmail.com>
 <d426e74a-3d26-4bcb-98d1-dd6e797b07f3@email.android.com>
Message-ID: <CAD=7XjM_t9xfiKYYEN0uZNdEHicg8DTbP370nWy+dquvxMfRAg@mail.gmail.com>

Hi Armin,

Thanks for the help! I tried your suggestion, and I get the following error
when I try to call that function on a wrapped object (instead of
_my_extra_dict, I named the attribute "rbflags" and called the function
get_rbflags) :

Attribute 'rb_flags' on <ClassDef 'pypy.interpreter.baseobjspace.W_Root'>
should be read-only.
This error can be caused by another 'getattr' that promoted
the attribute here; the list of read locations is:
(pypy.interpreter.baseobjspace:39)W_Root.get_rbflags


    v1066 = setattr(self_275, ('rb_flags'), v1065)

In <FunctionGraph of (pypy.interpreter.baseobjspace:39)W_Root.get_rbflags
at 0x65b54e90>:
Happened at file
/home/ubuntu/pypy2-v5.3.1-src/pypy/interpreter/baseobjspace.py line 41

==>             self.rb_flags = {}

Known variable annotations:
 self_275 = SomeInstance(can_be_None=False,
classdef=pypy.interpreter.baseobjspace.W_Root)
 v1065 = SomeOrderedDict(dictdef=<{SomeImpossibleValue():
SomeImpossibleValue()}>)

Processing block:
 block at 15 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
 in (pypy.interpreter.baseobjspace:39)W_Root.get_rbflags
 containing the following operations:
       v1065 = newdict()
       v1066 = setattr(self_275, ('rb_flags'), v1065)
 --end--

Frank

On Fri, Dec 16, 2016 at 2:20 PM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

> Hi Frank,
>
> Yes, you'll get an instance variable that way.
>
> Cheers,
>
> Carl Friedrich
>
> On December 16, 2016 6:05:56 PM GMT+01:00, Frank Wang <frankw at mit.edu>
> wrote:
>
>> Hi Armin,
>>
>> Thanks for the suggestion! I'll see if that works. Just to make sure.
>> This will give me an instance variable? I need values of the extra dict to
>> be different for different instantiations of W_Root.
>>
>> Frank
>>
>> On Fri, Dec 16, 2016 at 11:19 AM, Armin Rigo <armin.rigo at gmail.com>
>> wrote:
>>
>>> Hi Frank,
>>>
>>> On 15 December 2016 at 21:06, Frank Wang <frankw at mit.edu> wrote:
>>> > Right now, I know W_Root has no __init__ function, and when I try to
>>> add
>>> > one. I run into all sorts of problems because TypeDef calls
>>> W_Root.__new__
>>> > with some parameters, but "new" function seems to be called nowhere
>>> else.
>>>
>>> I suspect there is no clean way to add a __init__() method to W_Root.
>>> You could use default attributes instead:
>>>
>>> class W_Root:
>>>     _my_extra_dict = None
>>>
>>>     def get_extra_dict(self):
>>>         if self._my_extra_dict is None:
>>>             self._my_extra_dict = {}
>>>         return self._my_extra_dict
>>>
>>>
>>>
>>> A bient?t,
>>>
>>> Armin.
>>>
>>
>> ------------------------------
>>
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161218/5efa4e06/attachment.html>

From armin.rigo at gmail.com  Mon Dec 19 05:26:44 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Mon, 19 Dec 2016 11:26:44 +0100
Subject: [pypy-dev] Adding init/variables to W_Root
In-Reply-To: <CAD=7XjM_t9xfiKYYEN0uZNdEHicg8DTbP370nWy+dquvxMfRAg@mail.gmail.com>
References: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
 <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>
 <CAD=7XjN9XjAMYvotxU_6fZ6=D4+0y67RmZLDg=_tE-TkimSsZA@mail.gmail.com>
 <d426e74a-3d26-4bcb-98d1-dd6e797b07f3@email.android.com>
 <CAD=7XjM_t9xfiKYYEN0uZNdEHicg8DTbP370nWy+dquvxMfRAg@mail.gmail.com>
Message-ID: <CAMSv6X1rWf_Y+SzwL-upiJeBhtSXqDFm8s6XugVxHx2+3Af_yA@mail.gmail.com>

Hi Frank,

On 19 December 2016 at 05:29, Frank Wang <frankw at mit.edu> wrote:
> Attribute 'rb_flags' on <ClassDef 'pypy.interpreter.baseobjspace.W_Root'>
> should be read-only.

Ah, did you adapt or remove this line in class W_Root?

    __slots__ = ('__weakref__',)

The annotator complains, I think, because 'rb_flags' is forbidden by
this line (which is useful to avoid having random attributes
accidentally move up to W_Root).


A bient?t,

Armin.

From frankw at mit.edu  Mon Dec 19 11:42:12 2016
From: frankw at mit.edu (Frank Wang)
Date: Mon, 19 Dec 2016 08:42:12 -0800
Subject: [pypy-dev] Adding init/variables to W_Root
In-Reply-To: <CAMSv6X1rWf_Y+SzwL-upiJeBhtSXqDFm8s6XugVxHx2+3Af_yA@mail.gmail.com>
References: <CAD=7XjO_AG5KtLbiydCUXhROJkbJB7qwfEsuiXjGJrMRooFeog@mail.gmail.com>
 <CAMSv6X21brR2L7ZsUF1YVokpOxqmn_ZNZn0KQH-HzYfFDR=uuw@mail.gmail.com>
 <CAD=7XjN9XjAMYvotxU_6fZ6=D4+0y67RmZLDg=_tE-TkimSsZA@mail.gmail.com>
 <d426e74a-3d26-4bcb-98d1-dd6e797b07f3@email.android.com>
 <CAD=7XjM_t9xfiKYYEN0uZNdEHicg8DTbP370nWy+dquvxMfRAg@mail.gmail.com>
 <CAMSv6X1rWf_Y+SzwL-upiJeBhtSXqDFm8s6XugVxHx2+3Af_yA@mail.gmail.com>
Message-ID: <CAD=7XjNkLH3CiCToF7MAjL+MoPdbYyB2X2LipgxPhO9jmKxNqQ@mail.gmail.com>

Hi Armin,

If I modify the line to __slots__ = ('__weakref__', 'rb_flags',), I get the
following error:

[translation:ERROR] AssertionError:
<pypy.interpreter.generator.GeneratorIterator object at
0x00000000073e87c8>: to run register_finalizer() untranslated, the object
must not have __slots__

If I remove that line completely, I get the following error,

[translation:ERROR] Exception: <class
'pypy.objspace.std.dictmultiobject.W_DictMultiObject'> has slots or
_attrs_, but not its base class
Processing block:
 block at 3 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
 in (pypy.interpreter.mixedmodule:83)MixedModule.getdictvalue
 containing the following operations:
       v0 = getattr(space_0, ('finditem_str'))
       v1 = getattr(self_0, ('w_dict'))
       w_value_0 = simple_call(v0, v1, name_0)
       v2 = getattr(self_0, ('lazy'))
       v3 = bool(v2)
 --end--

I wonder what error in your opinion is easier to get around.

Thanks for the help!

Frank

On Mon, Dec 19, 2016 at 2:26 AM, Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi Frank,
>
> On 19 December 2016 at 05:29, Frank Wang <frankw at mit.edu> wrote:
> > Attribute 'rb_flags' on <ClassDef 'pypy.interpreter.
> baseobjspace.W_Root'>
> > should be read-only.
>
> Ah, did you adapt or remove this line in class W_Root?
>
>     __slots__ = ('__weakref__',)
>
> The annotator complains, I think, because 'rb_flags' is forbidden by
> this line (which is useful to avoid having random attributes
> accidentally move up to W_Root).
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161219/c89dea20/attachment.html>

From cfbolz at gmx.de  Mon Dec 19 12:00:12 2016
From: cfbolz at gmx.de (Carl Friedrich Bolz)
Date: Mon, 19 Dec 2016 18:00:12 +0100
Subject: [pypy-dev] Fwd: Re:  Adding init/variables to W_Root
In-Reply-To: <93774165-2ebb-492e-bc94-8366beefbd84@email.android.com>
References: <93774165-2ebb-492e-bc94-8366beefbd84@email.android.com>
Message-ID: <03a439ec-71fa-454a-a9e0-4238d7505513@email.android.com>

Oops, forgot to cc pypy-dev...

Carl Friedrich


-------- Original Message --------
From: Carl Friedrich Bolz <cfbolz at gmx.de>
Sent: December 19, 2016 5:59:01 PM GMT+01:00
To: Frank Wang <frankw at mit.edu>
Subject: Re: [pypy-dev] Adding init/variables to W_Root

Hi Frank, 

The solution is to add an _attrs_ = ('__weakref__', 'rb_flags',) declaration to the body of W_Root. It's like  __slots__, but only for RPython. 

Cheers, 
Carl Friedrich

On December 19, 2016 5:42:12 PM GMT+01:00, Frank Wang <frankw at mit.edu> wrote:
>Hi Armin,
>
>If I modify the line to __slots__ = ('__weakref__', 'rb_flags',), I get
>the
>following error:
>
>[translation:ERROR] AssertionError:
><pypy.interpreter.generator.GeneratorIterator object at
>0x00000000073e87c8>: to run register_finalizer() untranslated, the
>object
>must not have __slots__
>
>If I remove that line completely, I get the following error,
>
>[translation:ERROR] Exception: <class
>'pypy.objspace.std.dictmultiobject.W_DictMultiObject'> has slots or
>_attrs_, but not its base class
>Processing block:
> block at 3 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
> in (pypy.interpreter.mixedmodule:83)MixedModule.getdictvalue
> containing the following operations:
>       v0 = getattr(space_0, ('finditem_str'))
>       v1 = getattr(self_0, ('w_dict'))
>       w_value_0 = simple_call(v0, v1, name_0)
>       v2 = getattr(self_0, ('lazy'))
>       v3 = bool(v2)
> --end--
>
>I wonder what error in your opinion is easier to get around.
>
>Thanks for the help!
>
>Frank
>
>On Mon, Dec 19, 2016 at 2:26 AM, Armin Rigo <armin.rigo at gmail.com>
>wrote:
>
>> Hi Frank,
>>
>> On 19 December 2016 at 05:29, Frank Wang <frankw at mit.edu> wrote:
>> > Attribute 'rb_flags' on <ClassDef 'pypy.interpreter.
>> baseobjspace.W_Root'>
>> > should be read-only.
>>
>> Ah, did you adapt or remove this line in class W_Root?
>>
>>     __slots__ = ('__weakref__',)
>>
>> The annotator complains, I think, because 'rb_flags' is forbidden by
>> this line (which is useful to avoid having random attributes
>> accidentally move up to W_Root).
>>
>>
>> A bient?t,
>>
>> Armin.
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>pypy-dev mailing list
>pypy-dev at python.org
>https://mail.python.org/mailman/listinfo/pypy-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161219/7dcee352/attachment.html>

From frankw at mit.edu  Mon Dec 19 19:29:09 2016
From: frankw at mit.edu (Frank Wang)
Date: Mon, 19 Dec 2016 16:29:09 -0800
Subject: [pypy-dev] Fwd: Re: Adding init/variables to W_Root
In-Reply-To: <CAD=7XjOzwMMc+6TBL31s4xh71CKG-MtA7LQhqdCxuSw4G_zJXA@mail.gmail.com>
References: <93774165-2ebb-492e-bc94-8366beefbd84@email.android.com>
 <03a439ec-71fa-454a-a9e0-4238d7505513@email.android.com>
 <CAD=7XjOzwMMc+6TBL31s4xh71CKG-MtA7LQhqdCxuSw4G_zJXA@mail.gmail.com>
Message-ID: <CAD=7XjOLqAeuTqok0Pje18xFTVD1X8EjfjCY5Rf7kafxwEpnBw@mail.gmail.com>

This also seems to happen in a lot of classes because W_Root is the parent
class for many classes. Is there a way to universally turn off this
immutable flag (assuming it's okay) in all classes?

Frank

On Mon, Dec 19, 2016 at 3:55 PM, Frank Wang <frankw at mit.edu> wrote:

> Hi Carl,
>
> Thanks for the tip! It's mostly working. I get this error:
>
> [translation:ERROR] ImmutableConflictError: class <InstanceRepr for
> pypy.module._cffi_backend.libraryobj.W_Library> has _immutable_=True, but
> parent class <InstanceRepr for pypy.interpreter.baseobjspace.W_Root>
> defines (at least) the mutable field 'rb_flags'
>
> I'm tempted to set the flag to be False. Is there any reason that W_Root
> or any of its subclasses cannot contain immutable variables?
>
> Frank
>
> On Mon, Dec 19, 2016 at 9:00 AM, Carl Friedrich Bolz <cfbolz at gmx.de>
> wrote:
>
>> Oops, forgot to cc pypy-dev...
>>
>>
>> Carl Friedrich
>>
>> ------------------------------
>> *From:* Carl Friedrich Bolz <cfbolz at gmx.de>
>> *Sent:* December 19, 2016 5:59:01 PM GMT+01:00
>> *To:* Frank Wang <frankw at mit.edu>
>> *Subject:* Re: [pypy-dev] Adding init/variables to W_Root
>>
>> Hi Frank,
>>
>> The solution is to add an _attrs_ = ('__weakref__', 'rb_flags',)
>> declaration to the body of W_Root. It's like  __slots__, but only for
>> RPython.
>>
>> Cheers,
>>
>> Carl Friedrich
>>
>> On December 19, 2016 5:42:12 PM GMT+01:00, Frank Wang <frankw at mit.edu>
>> wrote:
>>>
>>> Hi Armin,
>>>
>>> If I modify the line to __slots__ = ('__weakref__', 'rb_flags',), I get
>>> the following error:
>>>
>>> [translation:ERROR] AssertionError: <pypy.interpreter.generator.GeneratorIterator
>>> object at 0x00000000073e87c8>: to run register_finalizer() untranslated,
>>> the object must not have __slots__
>>>
>>> If I remove that line completely, I get the following error,
>>>
>>> [translation:ERROR] Exception: <class 'pypy.objspace.std.dictmultiobject.W_DictMultiObject'>
>>> has slots or _attrs_, but not its base class
>>> Processing block:
>>>  block at 3 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>>>  in (pypy.interpreter.mixedmodule:83)MixedModule.getdictvalue
>>>  containing the following operations:
>>>        v0 = getattr(space_0, ('finditem_str'))
>>>        v1 = getattr(self_0, ('w_dict'))
>>>        w_value_0 = simple_call(v0, v1, name_0)
>>>        v2 = getattr(self_0, ('lazy'))
>>>        v3 = bool(v2)
>>>  --end--
>>>
>>> I wonder what error in your opinion is easier to get around.
>>>
>>> Thanks for the help!
>>>
>>> Frank
>>>
>>> On Mon, Dec 19, 2016 at 2:26 AM, Armin Rigo <armin.rigo at gmail.com>
>>> wrote:
>>>
>>>> Hi Frank,
>>>>
>>>> On 19 December 2016 at 05:29, Frank Wang <frankw at mit.edu> wrote:
>>>> > Attribute 'rb_flags' on <ClassDef 'pypy.interpreter.baseobjspace
>>>> .W_Root'>
>>>> > should be read-only.
>>>>
>>>> Ah, did you adapt or remove this line in class W_Root?
>>>>
>>>>     __slots__ = ('__weakref__',)
>>>>
>>>> The annotator complains, I think, because 'rb_flags' is forbidden by
>>>> this line (which is useful to avoid having random attributes
>>>> accidentally move up to W_Root).
>>>>
>>>>
>>>> A bient?t,
>>>>
>>>> Armin.
>>>>
>>>
>>> ------------------------------
>>>
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>>
>>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161219/29609be2/attachment.html>

From frankw at mit.edu  Mon Dec 19 18:55:40 2016
From: frankw at mit.edu (Frank Wang)
Date: Mon, 19 Dec 2016 15:55:40 -0800
Subject: [pypy-dev] Fwd: Re: Adding init/variables to W_Root
In-Reply-To: <03a439ec-71fa-454a-a9e0-4238d7505513@email.android.com>
References: <93774165-2ebb-492e-bc94-8366beefbd84@email.android.com>
 <03a439ec-71fa-454a-a9e0-4238d7505513@email.android.com>
Message-ID: <CAD=7XjOzwMMc+6TBL31s4xh71CKG-MtA7LQhqdCxuSw4G_zJXA@mail.gmail.com>

Hi Carl,

Thanks for the tip! It's mostly working. I get this error:

[translation:ERROR] ImmutableConflictError: class <InstanceRepr for
pypy.module._cffi_backend.libraryobj.W_Library> has _immutable_=True, but
parent class <InstanceRepr for pypy.interpreter.baseobjspace.W_Root>
defines (at least) the mutable field 'rb_flags'

I'm tempted to set the flag to be False. Is there any reason that W_Root or
any of its subclasses cannot contain immutable variables?

Frank

On Mon, Dec 19, 2016 at 9:00 AM, Carl Friedrich Bolz <cfbolz at gmx.de> wrote:

> Oops, forgot to cc pypy-dev...
>
>
> Carl Friedrich
>
> ------------------------------
> *From:* Carl Friedrich Bolz <cfbolz at gmx.de>
> *Sent:* December 19, 2016 5:59:01 PM GMT+01:00
> *To:* Frank Wang <frankw at mit.edu>
> *Subject:* Re: [pypy-dev] Adding init/variables to W_Root
>
> Hi Frank,
>
> The solution is to add an _attrs_ = ('__weakref__', 'rb_flags',)
> declaration to the body of W_Root. It's like  __slots__, but only for
> RPython.
>
> Cheers,
>
> Carl Friedrich
>
> On December 19, 2016 5:42:12 PM GMT+01:00, Frank Wang <frankw at mit.edu>
> wrote:
>>
>> Hi Armin,
>>
>> If I modify the line to __slots__ = ('__weakref__', 'rb_flags',), I get
>> the following error:
>>
>> [translation:ERROR] AssertionError: <pypy.interpreter.generator.GeneratorIterator
>> object at 0x00000000073e87c8>: to run register_finalizer() untranslated,
>> the object must not have __slots__
>>
>> If I remove that line completely, I get the following error,
>>
>> [translation:ERROR] Exception: <class 'pypy.objspace.std.
>> dictmultiobject.W_DictMultiObject'> has slots or _attrs_, but not its
>> base class
>> Processing block:
>>  block at 3 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>>  in (pypy.interpreter.mixedmodule:83)MixedModule.getdictvalue
>>  containing the following operations:
>>        v0 = getattr(space_0, ('finditem_str'))
>>        v1 = getattr(self_0, ('w_dict'))
>>        w_value_0 = simple_call(v0, v1, name_0)
>>        v2 = getattr(self_0, ('lazy'))
>>        v3 = bool(v2)
>>  --end--
>>
>> I wonder what error in your opinion is easier to get around.
>>
>> Thanks for the help!
>>
>> Frank
>>
>> On Mon, Dec 19, 2016 at 2:26 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
>>
>>> Hi Frank,
>>>
>>> On 19 December 2016 at 05:29, Frank Wang <frankw at mit.edu> wrote:
>>> > Attribute 'rb_flags' on <ClassDef 'pypy.interpreter.baseobjspace
>>> .W_Root'>
>>> > should be read-only.
>>>
>>> Ah, did you adapt or remove this line in class W_Root?
>>>
>>>     __slots__ = ('__weakref__',)
>>>
>>> The annotator complains, I think, because 'rb_flags' is forbidden by
>>> this line (which is useful to avoid having random attributes
>>> accidentally move up to W_Root).
>>>
>>>
>>> A bient?t,
>>>
>>> Armin.
>>>
>>
>> ------------------------------
>>
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>>
>>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161219/4626d9c8/attachment-0001.html>

From armin.rigo at gmail.com  Mon Dec 19 20:57:08 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Tue, 20 Dec 2016 02:57:08 +0100
Subject: [pypy-dev] Fwd: Re: Adding init/variables to W_Root
In-Reply-To: <CAD=7XjOLqAeuTqok0Pje18xFTVD1X8EjfjCY5Rf7kafxwEpnBw@mail.gmail.com>
References: <93774165-2ebb-492e-bc94-8366beefbd84@email.android.com>
 <03a439ec-71fa-454a-a9e0-4238d7505513@email.android.com>
 <CAD=7XjOzwMMc+6TBL31s4xh71CKG-MtA7LQhqdCxuSw4G_zJXA@mail.gmail.com>
 <CAD=7XjOLqAeuTqok0Pje18xFTVD1X8EjfjCY5Rf7kafxwEpnBw@mail.gmail.com>
Message-ID: <CAMSv6X110t+1iyWvoOt_UfKXrZu0oprYScikrh4+gPMD8wBhBA@mail.gmail.com>

Hi Frank,

On 20 December 2016 at 01:29, Frank Wang <frankw at mit.edu> wrote:
> This also seems to happen in a lot of classes because W_Root is the parent
> class for many classes. Is there a way to universally turn off this
> immutable flag (assuming it's okay) in all classes?

You can hack at the place where the error is raised.  Assuming you're
translating without the JIT, it should not cause too much problems.

As an alternative to adding an attribute to every object, you could
also use a rpython.rlib.rweakref.RWeakKeyDictionary(W_Root, RbFlags),
for some class RbFlags which has got the rb_flags dictionary and/or
any other related data.  The advantage is that it is a less intrusive
change, reduces memory usage if many objects don't need the extra
information at all, and avoids the immutable problem.  The
inconvenient is that going from the object to the rb_flags dictionary
is slower.

There are also alternatives that are better if you are ok with
supporting not *all* objects.  For example, you can edit
objspace/std/mapdict.py to add a new dynamic attribute that is
internally called ``"rbflags", SPECIAL``; see ``"weakref", SPECIAL``
for an existing example.  This would work for all objects that use
mapdicts, i.e. all instances of user-defined classes, and additionally
instances of many built-in types as well---but not instances of the
core types like "int" or "str" or "list" or "tuple" or "dict".


A bient?t,

Armin.

From f2013112 at goa.bits-pilani.ac.in  Mon Dec 19 07:03:50 2016
From: f2013112 at goa.bits-pilani.ac.in (George Joseph)
Date: Mon, 19 Dec 2016 17:33:50 +0530
Subject: [pypy-dev] Pypy Contribution
Message-ID: <CAPWevarYvAzcqCj8bCBWF0Vdfr93qSwTbwXyA1_ee1S6vYRkbw@mail.gmail.com>

Hi,
                  I'm an undergrad Computer Science student from India in
my final year. I'm interested in participating in GSoC 2017 for python, but
I'm new to open source. Would You kindly tell he how I can start
contributing? I'm familiar with C, C++, Java, Python, MATLAB, Verilog MySQL.

Thanks
?????????????????????????????????????????????
*George Joseph*
*M.Sc (Hons.) Physics + B.E. (Hons.) Computer Science*
*Birla Institute of Technology and Science, Pilani*
*K. K. Birla Goa Campus *
*+91 9503519766 | georgejoseph1 at protonmail.com
<georgejoseph1 at protonmail.com>*
?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161219/ade719de/attachment.html>

From frankw at mit.edu  Tue Dec 20 01:32:13 2016
From: frankw at mit.edu (Frank Wang)
Date: Tue, 20 Dec 2016 06:32:13 +0000
Subject: [pypy-dev] Fwd: Re: Adding init/variables to W_Root
In-Reply-To: <CAMSv6X110t+1iyWvoOt_UfKXrZu0oprYScikrh4+gPMD8wBhBA@mail.gmail.com>
References: <93774165-2ebb-492e-bc94-8366beefbd84@email.android.com>
 <03a439ec-71fa-454a-a9e0-4238d7505513@email.android.com>
 <CAD=7XjOzwMMc+6TBL31s4xh71CKG-MtA7LQhqdCxuSw4G_zJXA@mail.gmail.com>
 <CAD=7XjOLqAeuTqok0Pje18xFTVD1X8EjfjCY5Rf7kafxwEpnBw@mail.gmail.com>
 <CAMSv6X110t+1iyWvoOt_UfKXrZu0oprYScikrh4+gPMD8wBhBA@mail.gmail.com>
Message-ID: <CAD=7XjNuw_2j1vAAKJZGht-sMtoqJd4B_=wJzc485zfBWEierA@mail.gmail.com>

Hi Armin,

It seems that with your suggestions, things have started to work!

Thanks again for the help!
On Mon, Dec 19, 2016 at 5:57 PM Armin Rigo <armin.rigo at gmail.com> wrote:

> Hi Frank,
>
> On 20 December 2016 at 01:29, Frank Wang <frankw at mit.edu> wrote:
> > This also seems to happen in a lot of classes because W_Root is the
> parent
> > class for many classes. Is there a way to universally turn off this
> > immutable flag (assuming it's okay) in all classes?
>
> You can hack at the place where the error is raised.  Assuming you're
> translating without the JIT, it should not cause too much problems.
>
> As an alternative to adding an attribute to every object, you could
> also use a rpython.rlib.rweakref.RWeakKeyDictionary(W_Root, RbFlags),
> for some class RbFlags which has got the rb_flags dictionary and/or
> any other related data.  The advantage is that it is a less intrusive
> change, reduces memory usage if many objects don't need the extra
> information at all, and avoids the immutable problem.  The
> inconvenient is that going from the object to the rb_flags dictionary
> is slower.
>
> There are also alternatives that are better if you are ok with
> supporting not *all* objects.  For example, you can edit
> objspace/std/mapdict.py to add a new dynamic attribute that is
> internally called ``"rbflags", SPECIAL``; see ``"weakref", SPECIAL``
> for an existing example.  This would work for all objects that use
> mapdicts, i.e. all instances of user-defined classes, and additionally
> instances of many built-in types as well---but not instances of the
> core types like "int" or "str" or "list" or "tuple" or "dict".
>
>
> A bient?t,
>
> Armin.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161220/835f38dc/attachment.html>

From planrichi at gmail.com  Tue Dec 20 07:41:42 2016
From: planrichi at gmail.com (Richard Plangger)
Date: Tue, 20 Dec 2016 13:41:42 +0100
Subject: [pypy-dev] Pypy Contribution
In-Reply-To: <CAPWevarYvAzcqCj8bCBWF0Vdfr93qSwTbwXyA1_ee1S6vYRkbw@mail.gmail.com>
References: <CAPWevarYvAzcqCj8bCBWF0Vdfr93qSwTbwXyA1_ee1S6vYRkbw@mail.gmail.com>
Message-ID: <12cd5513-2c41-5707-360f-8e7c6e8809cf@gmail.com>

Hi,

I think a good approach is to start reading this page:
http://doc.pypy.org/en/latest/how-to-contribute.html

Generally speaking, we have lots of documentation at
http://doc.pypy.org, which you should use to get into PyPy development.

I think a solid approach would be to setup your development environment
and start to think about some ideas you could work on by changing some
details and reading code.

Our bugtracker at
https://bitbucket.org/pypy/pypy/issues?status=new&status=open might also
be a source for a task.

Cheers,
Richard

On 12/19/2016 01:03 PM, George Joseph wrote:
> Hi,
>                   I'm an undergrad Computer Science student from India
> in my final year. I'm interested in participating in GSoC 2017 for
> python, but I'm new to open source. Would You kindly tell he how I can
> start contributing? I'm familiar with C, C++, Java, Python, MATLAB,
> Verilog MySQL.
> 
> Thanks
> ?????????????????????????????????????????????
> *George Joseph*
> *M.Sc (Hons.) Physics + B.E. (Hons.) Computer Science*
> *Birla Institute of Technology and Science, Pilani*
> *K. K. Birla Goa Campus *
> *+91 9503519766 | georgejoseph1 at protonmail.com
> <mailto:georgejoseph1 at protonmail.com>*
> ?
> 
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161220/27e746f5/attachment-0001.sig>

From shubharamani at yahoo.com  Tue Dec 20 13:45:25 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Tue, 20 Dec 2016 18:45:25 +0000 (UTC)
Subject: [pypy-dev] which API returns Python stack in codemap.c given pc
 ?
In-Reply-To: <356395B0-0FE7-4C62-8C0F-C09375D82B15@yahoo.com>
References: <1687153522.872443.1482003121391.ref@mail.yahoo.com>
 <1687153522.872443.1482003121391@mail.yahoo.com>
 <CAMSv6X1V-tBubXo8UpujHzqK7ia8=eSUVQ13Sw+Wq7Z8e31Y4A@mail.gmail.com>
 <356395B0-0FE7-4C62-8C0F-C09375D82B15@yahoo.com>
Message-ID: <1113194482.2425265.1482259526085@mail.yahoo.com>

OK I see it now. vmprof_common.h shows how you can the program counter (PC) fromcodemap?pypy_find_codemap_at_addr() from codemap.c 

    On Sunday, December 18, 2016 5:47 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 Hi Armin. Yes I've studied the codes you mention deeply. They are fairly straightforward to follow so thank you for the tips.
But I'm merely responding to a statement you made in the August 2016 thread. See below.?
https://mail.python.org/pipermail/pypy-dev/2016-August/014597.html
Thanks Shubharamani?
On Dec 18, 2016, at 2:33 AM, Armin Rigo <armin.rigo at gmail.com> wrote:


Hi Shubha,

On 17 December 2016 at 20:32, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:

I have studied rpython/jit/backend/llsupport/src/codemap.c.


I am nt finding the API which returns the Python stack in codemap.c.


There is no such knowledge as the "Python stack" inside the JIT. ?Keep
in mind that the JIT is generic. ?You have to study the (generic)
rpython.rlib.rvmprof and the (Python-specific) pypy.module._vmprof to
find out how together they reconstruct this information.


A bient?t,

Armin.



   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161220/3ab7c7c9/attachment.html>

From shubharamani at yahoo.com  Tue Dec 20 17:48:36 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Tue, 20 Dec 2016 22:48:36 +0000 (UTC)
Subject: [pypy-dev] access to PyFrame within x86 backend
References: <2066816014.2597127.1482274116034.ref@mail.yahoo.com>
Message-ID: <2066816014.2597127.1482274116034@mail.yahoo.com>

First of all let me say that all I've done for the past several days is read, read read.I have read the rpython and pypy documentation. I've read?bolz-tracing-jit-final.pdf as well as antocuni-phd-thesis.pdf. I also?feel that I comprehended most of what I read.
In studying the vmprof implementation there is still a gap in my comprehension.?How do I get access to PyFrame within assembly.py ?
VmProf dumps everything into a specially formatted binary file, embedded with variousmarkers and the function code objects among many other stuff, per PyFrame which gets executed.?This file later gets post-processed by the vmprof tool.
Suppose I do something similar to vmprof in order to obtain access to function code objects, i.e.?see below. But how do I get access to PyFrame in the x86 backend code at run time ? ?
_get_code = lambda frame, w_inputvalue, operr: frame.pycode ? ? ? ? ? ? ? ? ? ?_decorator = rvmprof.vmprof_execute_code("pypy", _get_code, W_Root) ? ? ? ? ? ?my_execute_frame = _decorator(PyFrame.execute_frame) ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?class __extend__(PyFrame): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? def execute_frame(self, w_inputvalue=None, operr=None): ? ? ? ? ? ? ? ? ? ?? ? ? ? # indirection for the optional arguments ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? return my_execute_frame(self, w_inputvalue, operr)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161220/1ed009c2/attachment.html>

From shubharamani at yahoo.com  Tue Dec 20 20:01:50 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Wed, 21 Dec 2016 01:01:50 +0000 (UTC)
Subject: [pypy-dev] access to PyFrame within x86 backend
In-Reply-To: <2066816014.2597127.1482274116034@mail.yahoo.com>
References: <2066816014.2597127.1482274116034.ref@mail.yahoo.com>
 <2066816014.2597127.1482274116034@mail.yahoo.com>
Message-ID: <1588420503.2642393.1482282110974@mail.yahoo.com>

Is using?/opt/pypy_latest/rpython/rlib/rvmprof/ traceback.py (with a callback option, as shown in test/test_traceback.py)a good idea - to be called within x86/assembler.py ? That's the direction I'm heading unless there are other suggestions.
eating humble pie Shubha 

    On Tuesday, December 20, 2016 2:48 PM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 First of all let me say that all I've done for the past several days is read, read read.I have read the rpython and pypy documentation. I've read?bolz-tracing-jit-final.pdf as well as antocuni-phd-thesis.pdf. I also?feel that I comprehended most of what I read.
In studying the vmprof implementation there is still a gap in my comprehension.?How do I get access to PyFrame within assembly.py ?
VmProf dumps everything into a specially formatted binary file, embedded with variousmarkers and the function code objects among many other stuff, per PyFrame which gets executed.?This file later gets post-processed by the vmprof tool.
Suppose I do something similar to vmprof in order to obtain access to function code objects, i.e.?see below. But how do I get access to PyFrame in the x86 backend code at run time ? ?
_get_code = lambda frame, w_inputvalue, operr: frame.pycode ? ? ? ? ? ? ? ? ? ?_decorator = rvmprof.vmprof_execute_code("pypy", _get_code, W_Root) ? ? ? ? ? ?my_execute_frame = _decorator(PyFrame.execute_frame) ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?class __extend__(PyFrame): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? def execute_frame(self, w_inputvalue=None, operr=None): ? ? ? ? ? ? ? ? ? ?? ? ? ? # indirection for the optional arguments ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? return my_execute_frame(self, w_inputvalue, operr)?

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161221/044dcc99/attachment-0001.html>

From shubharamani at yahoo.com  Wed Dec 21 16:25:27 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Wed, 21 Dec 2016 21:25:27 +0000 (UTC)
Subject: [pypy-dev] access to PyFrame within x86 backend
In-Reply-To: <1588420503.2642393.1482282110974@mail.yahoo.com>
References: <2066816014.2597127.1482274116034.ref@mail.yahoo.com>
 <2066816014.2597127.1482274116034@mail.yahoo.com>
 <1588420503.2642393.1482282110974@mail.yahoo.com>
Message-ID: <339204150.500363.1482355527486@mail.yahoo.com>

Sorry that code snippet came out screwed up:
_get_code = lambda frame, w_inputvalue, operr: frame.pycode ? ? ? ? ? ? ? ? ? ?_decorator = rvmprof.vmprof_execute_code("pypy", _get_code, W_Root) ? ? ? ? ? ?my_execute_frame = _decorator(PyFrame.execute_frame) ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?class __extend__(PyFrame): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? def execute_frame(self, w_inputvalue=None, operr=None): ? ? ? ? ? ? ? ? ? ?? ? ? ? # indirection for the optional arguments ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? return my_execute_frame(self, w_inputvalue, operr)? 

    On Tuesday, December 20, 2016 5:01 PM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 Is using?/opt/pypy_latest/rpython/rlib/rvmprof/ traceback.py (with a callback option, as shown in test/test_traceback.py)a good idea - to be called within x86/assembler.py ? That's the direction I'm heading unless there are other suggestions.
eating humble pie Shubha 

    On Tuesday, December 20, 2016 2:48 PM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 First of all let me say that all I've done for the past several days is read, read read.I have read the rpython and pypy documentation. I've read?bolz-tracing-jit-final.pdf as well as antocuni-phd-thesis.pdf. I also?feel that I comprehended most of what I read.
In studying the vmprof implementation there is still a gap in my comprehension.?How do I get access to PyFrame within assembly.py ?
VmProf dumps everything into a specially formatted binary file, embedded with variousmarkers and the function code objects among many other stuff, per PyFrame which gets executed.?This file later gets post-processed by the vmprof tool.
Suppose I do something similar to vmprof in order to obtain access to function code objects, i.e.?see below. But how do I get access to PyFrame in the x86 backend code at run time ? ?
_get_code = lambda frame, w_inputvalue, operr: frame.pycode ? ? ? ? ? ? ? ? ? ?_decorator = rvmprof.vmprof_execute_code("pypy", _get_code, W_Root) ? ? ? ? ? ?my_execute_frame = _decorator(PyFrame.execute_frame) ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?class __extend__(PyFrame): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? def execute_frame(self, w_inputvalue=None, operr=None): ? ? ? ? ? ? ? ? ? ?? ? ? ? # indirection for the optional arguments ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? return my_execute_frame(self, w_inputvalue, operr)?

   

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161221/4a7194dd/attachment.html>

From kunshan.wang at anu.edu.au  Wed Dec 21 23:56:17 2016
From: kunshan.wang at anu.edu.au (Kunshan Wang)
Date: Thu, 22 Dec 2016 15:56:17 +1100
Subject: [pypy-dev] Object pinning
Message-ID: <6be3364c-d1aa-79f3-bb83-9b171562160a@anu.edu.au>

Hi folks,

I have a question regarding object pinning in RPython.

Consider the following snippet from rpython/rtyper/lltypesystem/rstr.py

    @jit.oopspec('stroruni.copy_contents(src, dst, srcstart, dststart,
length)')
    @signature(types.any(), types.any(), types.int(), types.int(),
types.int(), returns=types.none())
    def copy_string_contents(src, dst, srcstart, dststart, length):
        """Copies 'length' characters from the 'src' string to the 'dst'
        string, starting at position 'srcstart' and 'dststart'."""
        # xxx Warning: don't try to do this at home.  It relies on a lot
        # of details to be sure that it works correctly in all cases.
        # Notably: no GC operation at all from the first cast_ptr_to_adr()
        # because it might move the strings.  The keepalive_until_here()
        # are obscurely essential to make sure that the strings stay alive
        # longer than the raw_memcopy().
        assert length >= 0
        ll_assert(srcstart >= 0, "copystrc: negative srcstart")
        ll_assert(srcstart + length <= len(src.chars), "copystrc: src ovf")
        ll_assert(dststart >= 0, "copystrc: negative dststart")
        ll_assert(dststart + length <= len(dst.chars), "copystrc: dst ovf")
        # from here, no GC operations can happen
        asrc = _get_raw_buf(SRC_TP, src, srcstart)
        adst = _get_raw_buf(DST_TP, dst, dststart)
        llmemory.raw_memcopy(asrc, adst, llmemory.sizeof(CHAR_TP) * length)
        # end of "no GC" section
        keepalive_until_here(src)
        keepalive_until_here(dst)
    copy_string_contents._always_inline_ = True
    copy_string_contents = func_with_new_name(copy_string_contents,
                                              'copy_%s_contents' % name)

There is a region where heap objects in the RPython heap is accessed
externally by native programs.  I understand that GC must neither
recycle the object nor move it in the memory.  But I have two questions
about how object pinning is done in RPython:

(1) From the perspective of the RPython user (e.g. high-level language
implementer, interpreter writer, library writer, ...), what is the
"protocol" to follow when interacting with native programs (such as
passing a buffer to the `read` syscall)?  I have seen idiomatic use of
`cast_ptr_to_adr` followed by `keepalive_until_here`.  But there is also
`pin` and `unpin` functions in the rpython/rlib/rgc.py module.  What is
the expected way for *the user* to pin objects for native access?

(2) From the perspective of the RPython developer (those who develop the
translation from RTyped CFGs to C source code, assembly code and machine
code), how does the C backend actually enforce the no-GC policy between
"from here, no GC operations can happen" and "end of 'no GC' section"?
As I observed, keepalive_until_here essentially generates a no-op inline
assembly that "uses" the variable so that the C compiler keeps that
variable alive.  But what is preventing GC from happening?


Some background: We are building a new backend for RPython on the Mu
micro virtual machine (https://gitlab.anu.edu.au/mu/mu-client-pypy).
This VM has built-in GC and exception handling, so they don't need to be
injected after RTyper.  But the micro VM also keeps the representation
of object references opaque.  The only way to "cast" references to
addresses is using the "object pinning" operation which returns its
address.  The idiom in Mu when passing a buffer to native programs is
that you "pin" the object, give the pointer to the native functions
(such as `memcpy`, `read` and `write`), and then "unpin" it (nested
pinning is allowed, and it needs to be unpinned as many times as it was
pinned).  The GC will neither reclaim the object nor move it when it is
pinned, but object pinning does not prevent GC from happening: GC can
still move other objects, but not the pinned ones.

So the crux is how to translate the RPython primitives into the Mu
counterparts.  If `cast_ptr_to_adr` and `keep_alive_until_here` is a
well-obeyed idiom, we can simply translate them to `pin` and `unpin`,
respectively.

The problem is, it only works for non-GC types.  Mu cannot copy
references in this way.  There are several problems: (1) Mu does not
want to expose the byte-by-byte representation of references.  So
references may not really be addresses, and may not be copied naively as
a word (it could contain bit flags, too).  (2) Mu does not prevent the
movement of other object. So if an array of references is pinned, the
objects its elements point *to* may still be moved.  I guess the reason
why the snippet above prevents *all* kinds of GC activity is because it
may be used to copy GC-managed object references, too.  If this is the
case, we have to find alternative solutions.

Regards,
Kunshan Wang
School of Computer Science
Australian National University

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 496 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161222/e7e5e4b3/attachment-0001.sig>

From armin.rigo at gmail.com  Thu Dec 22 12:15:24 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Thu, 22 Dec 2016 18:15:24 +0100
Subject: [pypy-dev] Object pinning
In-Reply-To: <6be3364c-d1aa-79f3-bb83-9b171562160a@anu.edu.au>
References: <6be3364c-d1aa-79f3-bb83-9b171562160a@anu.edu.au>
Message-ID: <CAMSv6X0rUVtSjzM8_Nv1=-2CQhLAp=RqRG9xP7fYPM4ZPV6AEw@mail.gmail.com>

Hi Kunshan,

On 22 December 2016 at 05:56, Kunshan Wang <kunshan.wang at anu.edu.au> wrote:
> So the crux is how to translate the RPython primitives into the Mu
> counterparts.  If `cast_ptr_to_adr` and `keep_alive_until_here` is a
> well-obeyed idiom, we can simply translate them to `pin` and `unpin`,
> respectively.

No, I think what you need in this case is similar to what we have in
the stmgc-c8 branch.  In this branch, we are more careful than in
trunk about casting GC references to raw addresses; and we're also
using a separate GC (implemented in C) which may be closer to what
you're trying to do as well.

In this branch, when we translate an RPython program with --stm, we
have no notion of casting a GC reference to a raw address at all: it
is just forbidden.  The problem with stm is that a GC reference is not
really a raw address.  There are actually ways in which we can still
obtain the raw address if we're extra extra careful, but the default
casts explicitly don't work.  So for code like copy_string_contents(),
the branch adds the following (this example in the
copy_string_contents() function):

        # STM requires the slow character-by-character version,
        # which at least works without forcing the transaction
        # to become inevitable
        if rgc.stm_is_enabled():
            i = 0
            while i < length:
                dst.chars[dststart + i] = src.chars[srcstart + i]
                i += 1
            return

These check remove *all* casts.  For example, the RPython os.write()
by default works by trying to pin temporarily the string; but with
--stm there is no pinning and what occurs is that it always copies the
string into a raw-malloced buffer.

You can of course later think about how to avoid some of these copies
by re-introducing some pinning, but the above should allow you to
progress.


A bient?t,

Armin.

From shubharamani at yahoo.com  Thu Dec 22 17:58:12 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 22 Dec 2016 22:58:12 +0000 (UTC)
Subject: [pypy-dev] [translation:ERROR] Exception: 'no_release_gil' function
 can release the GIL
References: <241745154.611596.1482447492979.ref@mail.yahoo.com>
Message-ID: <241745154.611596.1482447492979@mail.yahoo.com>

Why ? I cannot seem to open a file anywhere within assembly.py. When I use "NOT_RPYTHON"in method open_file then I cannot use it in functions like assemble_loop because it's NOT_RPYTHON.
Happens when I call the following function in x86/assembly.py:?
def open_file(self): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? self.vtune_file = open('.\vtune.log', 'w') ??
self.fixup_target_tokens(rawstart) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??self.materialize_done(rawstart, full_size, "loop%d" % looptoken.number) ? ? ? ?self.open_file()


[translation:ERROR] Exception: 'no_release_gil' function can release the GIL: <function assemble_loop at 0x0000000070d4d2e0>[GilAnalyzer] analyze_direct_call((rpython.rtyper.lltypesystem.rffi:3)ccall_fopen__arrayPtr_arrayPtr): True[GilAnalyzer] analyze_direct_call((rpython.rtyper.lltypesystem.rffi:236)fopen__arrayPtr_arrayPtr_star_2): True[GilAnalyzer] analyze_direct_call((rpython.rlib.rfile:154)create_file): True[translation] start debugger...> /opt/vtune_pypy/rpython/translator/backendopt/gilanalysis.py(51)analyze()-> " %s\n%s" % (func, err.getvalue()))(Pdb+) print func<function assemble_loop at 0x0000000070d4d2e0>(Pdb+)?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161222/c736e407/attachment.html>

From shubharamani at yahoo.com  Thu Dec 22 19:07:31 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 23 Dec 2016 00:07:31 +0000 (UTC)
Subject: [pypy-dev] [translation:ERROR] Exception: 'no_release_gil'
 function can release the GIL
In-Reply-To: <241745154.611596.1482447492979@mail.yahoo.com>
References: <241745154.611596.1482447492979.ref@mail.yahoo.com>
 <241745154.611596.1482447492979@mail.yahoo.com>
Message-ID: <1862749653.628844.1482451651994@mail.yahoo.com>

Same error with this version :def open_vtune_file(self): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? from rpython.rlib.streamio import open_file_as_stream ? ? ? ? ? ? ? ? ? ? ? ? ?? ? ? ? self.vtune_file = open_file_as_stream('.\vtune.log', 'w')? 

    On Thursday, December 22, 2016 2:58 PM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 Why ? I cannot seem to open a file anywhere within assembly.py. When I use "NOT_RPYTHON"in method open_file then I cannot use it in functions like assemble_loop because it's NOT_RPYTHON.
Happens when I call the following function in x86/assembly.py:?
def open_file(self): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? self.vtune_file = open('.\vtune.log', 'w') ??
self.fixup_target_tokens(rawstart) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??self.materialize_done(rawstart, full_size, "loop%d" % looptoken.number) ? ? ? ?self.open_file()


[translation:ERROR] Exception: 'no_release_gil' function can release the GIL: <function assemble_loop at 0x0000000070d4d2e0>[GilAnalyzer] analyze_direct_call((rpython.rtyper.lltypesystem.rffi:3)ccall_fopen__arrayPtr_arrayPtr): True[GilAnalyzer] analyze_direct_call((rpython.rtyper.lltypesystem.rffi:236)fopen__arrayPtr_arrayPtr_star_2): True[GilAnalyzer] analyze_direct_call((rpython.rlib.rfile:154)create_file): True[translation] start debugger...> /opt/vtune_pypy/rpython/translator/backendopt/gilanalysis.py(51)analyze()-> " %s\n%s" % (func, err.getvalue()))(Pdb+) print func<function assemble_loop at 0x0000000070d4d2e0>(Pdb+)?




   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161223/d2ccd9c7/attachment.html>

From william.leslie.ttg at gmail.com  Thu Dec 22 21:57:06 2016
From: william.leslie.ttg at gmail.com (William ML Leslie)
Date: Fri, 23 Dec 2016 13:57:06 +1100
Subject: [pypy-dev] Object pinning
In-Reply-To: <6be3364c-d1aa-79f3-bb83-9b171562160a@anu.edu.au>
References: <6be3364c-d1aa-79f3-bb83-9b171562160a@anu.edu.au>
Message-ID: <CAHgd1hGbrw7WGuDvshPU5x-=zRr2hbjBQ62q-3ZeGy0qmSivsg@mail.gmail.com>

On 22 December 2016 at 15:56, Kunshan Wang <kunshan.wang at anu.edu.au> wrote:
> Hi folks,

Hi Kunshan!

>
> I have a question regarding object pinning in RPython.
>
> Consider the following snippet from rpython/rtyper/lltypesystem/rstr.py
>
>     @jit.oopspec('stroruni.copy_contents(src, dst, srcstart, dststart,
> length)')
>     @signature(types.any(), types.any(), types.int(), types.int(),
> types.int(), returns=types.none())
>     def copy_string_contents(src, dst, srcstart, dststart, length):
>         """Copies 'length' characters from the 'src' string to the 'dst'
>         string, starting at position 'srcstart' and 'dststart'."""
>         # xxx Warning: don't try to do this at home.  It relies on a lot
>         # of details to be sure that it works correctly in all cases.
>         # Notably: no GC operation at all from the first cast_ptr_to_adr()
>         # because it might move the strings.  The keepalive_until_here()
>         # are obscurely essential to make sure that the strings stay alive
>         # longer than the raw_memcopy().
>         assert length >= 0
>         ll_assert(srcstart >= 0, "copystrc: negative srcstart")
>         ll_assert(srcstart + length <= len(src.chars), "copystrc: src ovf")
>         ll_assert(dststart >= 0, "copystrc: negative dststart")
>         ll_assert(dststart + length <= len(dst.chars), "copystrc: dst ovf")
>         # from here, no GC operations can happen
>         asrc = _get_raw_buf(SRC_TP, src, srcstart)
>         adst = _get_raw_buf(DST_TP, dst, dststart)
>         llmemory.raw_memcopy(asrc, adst, llmemory.sizeof(CHAR_TP) * length)
>         # end of "no GC" section
>         keepalive_until_here(src)
>         keepalive_until_here(dst)
>     copy_string_contents._always_inline_ = True
>     copy_string_contents = func_with_new_name(copy_string_contents,
>                                               'copy_%s_contents' % name)
>
> There is a region where heap objects in the RPython heap is accessed
> externally by native programs.  I understand that GC must neither
> recycle the object nor move it in the memory.  But I have two questions
> about how object pinning is done in RPython:
>
> (1) From the perspective of the RPython user (e.g. high-level language
> implementer, interpreter writer, library writer, ...), what is the
> "protocol" to follow when interacting with native programs (such as
> passing a buffer to the `read` syscall)?  I have seen idiomatic use of
> `cast_ptr_to_adr` followed by `keepalive_until_here`.  But there is also
> `pin` and `unpin` functions in the rpython/rlib/rgc.py module.  What is
> the expected way for *the user* to pin objects for native access?
>

The general practice so far has been not to expose
interpreter-allocated objects to native code at all, but to give
native code either a raw malloc'd handle (a-la
https://bitbucket.org/pypy/pypy/src/29df4aac463be1c3508b47754014844edb765bfa/pypy/module/cpyext/pyobject.py?at=default&fileviewer=file-view-default#pyobject.py-52
) or require the native side to do the allocation (eg
https://bitbucket.org/pypy/pypy/src/29df4aac463be1c3508b47754014844edb765bfa/pypy/module/_cffi_backend/cbuffer.py?at=default&fileviewer=file-view-default
).  The aim was to significantly reduce the amount of memory that
needed pinning.

To the extent that there is an interface, it's the rpython.rlib.rgc
module.  A fair bit of lltypesystem code makes use of the assumption
that no other code is touching interpreter objects and the
implementation detail that if no heap memory is allocated in some
dynamic extent, the GC cannot run and move objects.  Seems a bit
disquieting.

TAN: you might have better primitives for dealing with strings in your
target, in which case avoiding much of the lltypesystem might be a
better deal for you.

> (2) From the perspective of the RPython developer (those who develop the
> translation from RTyped CFGs to C source code, assembly code and machine
> code), how does the C backend actually enforce the no-GC policy between
> "from here, no GC operations can happen" and "end of 'no GC' section"?
> As I observed, keepalive_until_here essentially generates a no-op inline
> assembly that "uses" the variable so that the C compiler keeps that
> variable alive.  But what is preventing GC from happening?
>

There are no instructions that may allocate within that dynamic
section.  It's not machine verified at all.  It could be.

>
> Some background: We are building a new backend for RPython on the Mu
> micro virtual machine (https://gitlab.anu.edu.au/mu/mu-client-pypy).
> This VM has built-in GC and exception handling, so they don't need to be
> injected after RTyper.  But the micro VM also keeps the representation
> of object references opaque.  The only way to "cast" references to
> addresses is using the "object pinning" operation which returns its
> address.  The idiom in Mu when passing a buffer to native programs is
> that you "pin" the object, give the pointer to the native functions
> (such as `memcpy`, `read` and `write`), and then "unpin" it (nested
> pinning is allowed, and it needs to be unpinned as many times as it was
> pinned).  The GC will neither reclaim the object nor move it when it is
> pinned, but object pinning does not prevent GC from happening: GC can
> still move other objects, but not the pinned ones.
>

Sounds sensible.

We have enough people playing with pypy in Au now that we should
probably all meet at some point (:

> So the crux is how to translate the RPython primitives into the Mu
> counterparts.  If `cast_ptr_to_adr` and `keep_alive_until_here` is a
> well-obeyed idiom, we can simply translate them to `pin` and `unpin`,
> respectively.
>
> The problem is, it only works for non-GC types.  Mu cannot copy
> references in this way.  There are several problems: (1) Mu does not
> want to expose the byte-by-byte representation of references.  So
> references may not really be addresses, and may not be copied naively as
> a word (it could contain bit flags, too).  (2) Mu does not prevent the
> movement of other object. So if an array of references is pinned, the
> objects its elements point *to* may still be moved.  I guess the reason
> why the snippet above prevents *all* kinds of GC activity is because it
> may be used to copy GC-managed object references, too.  If this is the
> case, we have to find alternative solutions.
>

Right, we have the same concern in pypy.  See cffi and cpyext for how
we hand such references to external libraries.

-- 
William Leslie

Notice:
Likely much of this email is, by the nature of copyright, covered
under copyright law.  You absolutely MAY reproduce any part of it in
accordance with the copyright law of the nation you are reading this
in.  Any attempt to DENY YOU THOSE RIGHTS would be illegal without
prior contractual agreement.

From armin.rigo at gmail.com  Fri Dec 23 05:12:41 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 23 Dec 2016 11:12:41 +0100
Subject: [pypy-dev] [translation:ERROR] Exception: 'no_release_gil'
 function can release the GIL
In-Reply-To: <241745154.611596.1482447492979@mail.yahoo.com>
References: <241745154.611596.1482447492979.ref@mail.yahoo.com>
 <241745154.611596.1482447492979@mail.yahoo.com>
Message-ID: <CAMSv6X3EvWCH79qTqmQw9Lkw7urgMgGgJKCmaHr9ULRpeKfgeg@mail.gmail.com>

Hi,

On 22 December 2016 at 23:58, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:
> [translation:ERROR] Exception: 'no_release_gil' function can release the
> GIL: <function assemble_loop at 0x0000000070d4d2e0>

assemble_loop() is marked as no_release_gil because things can get
subtly wrong if it releases the GIL in the middle of making machine
code and another thread reaches assemble_loop() too.

For debugging output, use debug_print().

If you really want to write a custom file, then you need to define and
use the C functions open(), write() and close() in a way that calling
them won't release the GIL.  Alternatively, you can remove
@rgc.no_release_gil on assemble_loop() and hope that the crashes you
get are very rare.

Or, stick everything on the AsmInfo() instance returned by
assemble_loop(), and dump that info later.


A bient?t,

Armin.

From armin.rigo at gmail.com  Sat Dec 24 04:54:57 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Sat, 24 Dec 2016 10:54:57 +0100
Subject: [pypy-dev] 3.6?
Message-ID: <CAMSv6X3B50q2Ay+iTvamRF9bk_zh45n4Oa_JrZuwpa4bnht9Fg@mail.gmail.com>

Hi Philip,

Can we get a discussion with respect to Python 3.6?  I see that you
created a "py3.6" branch.  However, the people funded by Mozilla (like
me) are more interested in finishing a good 3.5.x version first.  (I
suppose it's up to discussion if we decide to go for 3.6 immediately,
but I guess the answer is no.)

That means we'll keep releasing 3.5 versions and won't look into 3.6
before next summer.  It's unclear to me that it's worthwhile to start
a parallel py3.6 branch right now, as it'll just be merging hell for
the next 6 months.  Well, I suppose it's your hell, so I can't
complain on my own :-)

Just to say, maybe your time would be better spent helping us with
3.5, instead of maintaining py3.6 when py3.5 is still fast-moving.


A bient?t,

Armin.

From pjenvey at underboss.org  Sat Dec 24 13:17:07 2016
From: pjenvey at underboss.org (Philip Jenvey)
Date: Sat, 24 Dec 2016 10:17:07 -0800
Subject: [pypy-dev] 3.6?
In-Reply-To: <CAMSv6X3B50q2Ay+iTvamRF9bk_zh45n4Oa_JrZuwpa4bnht9Fg@mail.gmail.com>
References: <CAMSv6X3B50q2Ay+iTvamRF9bk_zh45n4Oa_JrZuwpa4bnht9Fg@mail.gmail.com>
Message-ID: <F0DC834A-0D60-4459-8862-BD42B5685B75@underboss.org>


> On Dec 24, 2016, at 1:54 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
> 
> Hi Philip,
> 
> Can we get a discussion with respect to Python 3.6?  I see that you
> created a "py3.6" branch.  However, the people funded by Mozilla (like
> me) are more interested in finishing a good 3.5.x version first.  (I
> suppose it's up to discussion if we decide to go for 3.6 immediately,
> but I guess the answer is no.)
> 
> That means we'll keep releasing 3.5 versions and won't look into 3.6
> before next summer.  It's unclear to me that it's worthwhile to start
> a parallel py3.6 branch right now, as it'll just be merging hell for
> the next 6 months.  Well, I suppose it's your hell, so I can't
> complain on my own :-)

Hey,

I?m not intending to do much more with it, I would rather help w/ 3.5 like you?re suggesting.

In the past there?s always been others who were eager to hack on the latest 3.x, so somebody was bound to start this branch at some point anyway. I figured I?d do the work of the merge now, and carefully ? we?ve unfortunately had some issues with this large 3.x vendor merge in the past =]

I also got a quick look at 3.6 and the gratification of more closely chasing the head of cpython after all these years.

?
Philip Jenvey

From phyo.arkarlwin at gmail.com  Sun Dec 25 02:50:45 2016
From: phyo.arkarlwin at gmail.com (Phyo Arkar)
Date: Sun, 25 Dec 2016 07:50:45 +0000
Subject: [pypy-dev] 3.6?
In-Reply-To: <F0DC834A-0D60-4459-8862-BD42B5685B75@underboss.org>
References: <CAMSv6X3B50q2Ay+iTvamRF9bk_zh45n4Oa_JrZuwpa4bnht9Fg@mail.gmail.com>
 <F0DC834A-0D60-4459-8862-BD42B5685B75@underboss.org>
Message-ID: <CA+HjJrhOeJeO9V_rc_NHcd-RAOGLvG3utEVhm9kNSEF8S1RFgg@mail.gmail.com>

Thank you very much on working towards 3.5 guys.
As users we can forget about 3.6 for now since 3.5 gives a lot of
improvment that most of us are looking for (asyn await and co ).
Would like to see it released soon , PyPy 3.5 is only thing we are waiting
, to move forward from python 2.7.x


On Sun, Dec 25, 2016 at 12:47 AM Philip Jenvey <pjenvey at underboss.org>
wrote:

>
> > On Dec 24, 2016, at 1:54 AM, Armin Rigo <armin.rigo at gmail.com> wrote:
> >
> > Hi Philip,
> >
> > Can we get a discussion with respect to Python 3.6?  I see that you
> > created a "py3.6" branch.  However, the people funded by Mozilla (like
> > me) are more interested in finishing a good 3.5.x version first.  (I
> > suppose it's up to discussion if we decide to go for 3.6 immediately,
> > but I guess the answer is no.)
> >
> > That means we'll keep releasing 3.5 versions and won't look into 3.6
> > before next summer.  It's unclear to me that it's worthwhile to start
> > a parallel py3.6 branch right now, as it'll just be merging hell for
> > the next 6 months.  Well, I suppose it's your hell, so I can't
> > complain on my own :-)
>
> Hey,
>
> I?m not intending to do much more with it, I would rather help w/ 3.5 like
> you?re suggesting.
>
> In the past there?s always been others who were eager to hack on the
> latest 3.x, so somebody was bound to start this branch at some point
> anyway. I figured I?d do the work of the merge now, and carefully ? we?ve
> unfortunately had some issues with this large 3.x vendor merge in the past
> =]
>
> I also got a quick look at 3.6 and the gratification of more closely
> chasing the head of cpython after all these years.
>
> ?
> Philip Jenvey
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161225/c502af45/attachment.html>

From matti.picus at gmail.com  Tue Dec 27 16:42:35 2016
From: matti.picus at gmail.com (Matti Picus)
Date: Tue, 27 Dec 2016 23:42:35 +0200
Subject: [pypy-dev] bitbucket enbled clonebundles
Message-ID: <3da0133a-8a82-f615-9f74-e49995ee5efc@gmail.com>

Bitbucket has enabled a feature called clonebundles which it is claimed 
will make "hg clone" much faster, for hg clients version 3.7 minimum.

I have enabled the feature on the pypy repos on bitbucket, after erik 
from atlassian pointed out its existence.

For reference, the toggle is on this page (admin permissions required) 
https://bitbucket.org/account/user/pypy/features.

Since the feature is in beta, please let erik know if you notice a 
difference

Matti


From rymg19 at gmail.com  Tue Dec 27 17:15:53 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 27 Dec 2016 16:15:53 -0600
Subject: [pypy-dev] bitbucket enbled clonebundles
In-Reply-To: <3da0133a-8a82-f615-9f74-e49995ee5efc@gmail.com>
References: <3da0133a-8a82-f615-9f74-e49995ee5efc@gmail.com>
Message-ID: <CAO41-mNAVA+z-wF5d8CmsthUwFkU-6bcDDsKdX7tggNAVeCfog@mail.gmail.com>

Does it make pulls any faster? I have the PyPy repo, but I haven't pulled
in over a year and am scared of it burning my CPU...

--
Ryan (????)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://kirbyfan64.github.io/

On Dec 27, 2016 3:43 PM, "Matti Picus" <matti.picus at gmail.com> wrote:

> Bitbucket has enabled a feature called clonebundles which it is claimed
> will make "hg clone" much faster, for hg clients version 3.7 minimum.
>
> I have enabled the feature on the pypy repos on bitbucket, after erik from
> atlassian pointed out its existence.
>
> For reference, the toggle is on this page (admin permissions required)
> https://bitbucket.org/account/user/pypy/features.
>
> Since the feature is in beta, please let erik know if you notice a
> difference
>
> Matti
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161227/80dcb8d9/attachment.html>

From evzijst at atlassian.com  Tue Dec 27 19:49:21 2016
From: evzijst at atlassian.com (Erik van Zijst)
Date: Tue, 27 Dec 2016 16:49:21 -0800
Subject: [pypy-dev] bitbucket enbled clonebundles
In-Reply-To: <CAO41-mNAVA+z-wF5d8CmsthUwFkU-6bcDDsKdX7tggNAVeCfog@mail.gmail.com>
References: <3da0133a-8a82-f615-9f74-e49995ee5efc@gmail.com>
 <CAO41-mNAVA+z-wF5d8CmsthUwFkU-6bcDDsKdX7tggNAVeCfog@mail.gmail.com>
Message-ID: <CAFUk6TAbxEqHScjiy3sni2bqWWLVU3=KeNKwMc9RG3c3JKA4Sw@mail.gmail.com>

On Tue, Dec 27, 2016 at 2:15 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> Does it make pulls any faster? I have the PyPy repo, but I haven't pulled in
> over a year and am scared of it burning my CPU...

No. Clonebundles only apply to the hg clone command. Pulls tend to be
small-ish and wouldn't typically benefit from fetching a bundle from a
CDN.

Cheers,
Erik


> Ryan (????)
> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
> http://kirbyfan64.github.io/
>
> On Dec 27, 2016 3:43 PM, "Matti Picus" <matti.picus at gmail.com> wrote:
>>
>> Bitbucket has enabled a feature called clonebundles which it is claimed
>> will make "hg clone" much faster, for hg clients version 3.7 minimum.
>>
>> I have enabled the feature on the pypy repos on bitbucket, after erik from
>> atlassian pointed out its existence.
>>
>> For reference, the toggle is on this page (admin permissions required)
>> https://bitbucket.org/account/user/pypy/features.
>>
>> Since the feature is in beta, please let erik know if you notice a
>> difference
>>
>> Matti
>>
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev

From armin.rigo at gmail.com  Wed Dec 28 03:48:07 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Wed, 28 Dec 2016 09:48:07 +0100
Subject: [pypy-dev] RPython Darwin platform description so_prefixes empty
In-Reply-To: <73AED7CF-06D8-477D-B15D-7CE5D03A512F@anu.edu.au>
References: <5053B1B0-2CF6-445F-83D2-005B14E26A92@anu.edu.au>
 <CAMSv6X2RAVs4rM7wgCddYn=oiCBjQZ+x-v9KrrbWzk4ucD_Q+g@mail.gmail.com>
 <FF79FCAC-939C-49DD-8EDF-0DE41F113E6F@anu.edu.au>
 <CAMSv6X3+FjOXvgimPk=Au78WabKRBt2zDSxiH6S8pSxizs_Y6A@mail.gmail.com>
 <73AED7CF-06D8-477D-B15D-7CE5D03A512F@anu.edu.au>
Message-ID: <CAMSv6X0CsQxgpsecUy4-iyLbOorjm_zuxjKLJhw2mjRi8tMA_w@mail.gmail.com>

Hi John,

On 7 December 2016 at 02:06, John Zhang <John.Zhang at anu.edu.au> wrote:
>> I think we can include 'lib' in so_prefix of all Posix platforms, is
>> that right?
>
> I can?t seem to find a definitive answer for all POSIX platforms. But for
> all the ones that PyPy currently supports, I think the answer is true. So if
> you want to make the changes then I think it?s a reasonable assumption and a
> good idea.

Checked in 4b1508ade877.

> On the other hand, I wonder if it?s also reasonable to modify the
> `_finish_linking` function to include `platform.so_prefix[0]` to the
> generated shared library name (see the attached patch).

It might or might not break stuff.  ``.so`` files without the ``lib``
prefix exist and are generally useful.  As far as it works this way,
I'd be wary of changes.


A bient?t,

Armin.

From shubharamani at yahoo.com  Thu Dec 29 12:14:17 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 29 Dec 2016 17:14:17 +0000 (UTC)
Subject: [pypy-dev] Debug build to analyze core dumps ?
References: <807366974.3320388.1483031657635.ref@mail.yahoo.com>
Message-ID: <807366974.3320388.1483031657635@mail.yahoo.com>

How do I build with debug symbols (i.e. -g) so that I can inspect core dumps when you get a traceback like this ?Do you always recompile with??pypy ../../rpython/bin/rpython --opt=jit ?
RPython traceback:? File "implement_11.c", line 5310, in fastfunc_enable_1? File "rpython_rlib_rvtune.c", line 1761, in VTune_enable? File "rpython_rlib_rvtune.c", line 2205, in gather_all_code_objs_1? File "rpython_rlib_rvtune.c", line 534, in VTune__write_code_registrationFatal RPython error: AssertionErrorAborted (core dumped)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161229/95c51329/attachment.html>

From shubharamani at yahoo.com  Thu Dec 29 14:18:34 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 29 Dec 2016 19:18:34 +0000 (UTC)
Subject: [pypy-dev] cannot use "id" in Rpython
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
Message-ID: <1704861386.3369487.1483039114642@mail.yahoo.com>


What can I use then ? I need the address of the function ?
[translation:ERROR] Exception: cannot use id() in RPython; see objectmodel.compute_xxx()Processing block:?block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>??in (pypy.module._vtune.interp_vtune:28)_get_full_name??containing the following operations:?? ? ? ?buf_0 = simple_call((type RStringIO))?? ? ? ?v943 = getattr(buf_0, ('write'))?? ? ? ?v944 = getattr(pycode_0, ('co_code'))?? ? ? ?v945 = simple_call(v943, v944)?? ? ? ?v946 = getattr(buf_0, ('getsize'))?? ? ? ?size_code_0 = simple_call(v946)?? ? ? ?v947 = getattr(pycode_0, ('co_name'))?? ? ? ?id_func_0 = id(v947)?? ? ? ?v948 = getattr(pycode_0, ('co_name'))?? ? ? ?v949 = simple_call((function _safe), v948)?? ? ? ?v950 = getattr(pycode_0, ('co_firstlineno'))?? ? ? ?v951 = getattr(pycode_0, ('co_filename'))?? ? ? ?v952 = simple_call((function _safe), v951)?? ? ? ?v953 = newtuple(v949, v950, v952, size_code_0, id_func_0)?? ? ? ?v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953)??--end--[translation] start debugger...> /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()-> raise Exception("cannot use id() in RPython; "(Pdb+)?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161229/30262e87/attachment.html>

From shubharamani at yahoo.com  Thu Dec 29 16:42:13 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Thu, 29 Dec 2016 21:42:13 +0000 (UTC)
Subject: [pypy-dev] Debug build to analyze core dumps ?
In-Reply-To: <807366974.3320388.1483031657635@mail.yahoo.com>
References: <807366974.3320388.1483031657635.ref@mail.yahoo.com>
 <807366974.3320388.1483031657635@mail.yahoo.com>
Message-ID: <1735157935.3443706.1483047734009@mail.yahoo.com>

Though I'm still curious about how to get debug builds so far the core dumps I've experienced are prettyeasy to root-cause. Attaching gdb seems overkill. 

    On Thursday, December 29, 2016 9:14 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 How do I build with debug symbols (i.e. -g) so that I can inspect core dumps when you get a traceback like this ?Do you always recompile with??pypy ../../rpython/bin/rpython --opt=jit ?
RPython traceback:? File "implement_11.c", line 5310, in fastfunc_enable_1? File "rpython_rlib_rvtune.c", line 1761, in VTune_enable? File "rpython_rlib_rvtune.c", line 2205, in gather_all_code_objs_1? File "rpython_rlib_rvtune.c", line 534, in VTune__write_code_registrationFatal RPython error: AssertionErrorAborted (core dumped)


   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161229/183a971e/attachment.html>

From rymg19 at gmail.com  Thu Dec 29 19:29:36 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 29 Dec 2016 18:29:36 -0600
Subject: [pypy-dev] cannot use "id" in Rpython
In-Reply-To: <1704861386.3369487.1483039114642@mail.yahoo.com>
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
 <1704861386.3369487.1483039114642@mail.yahoo.com>
Message-ID: <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>

What exactly are you trying to do. Using id to get the address of a
function is usually a bad idea...

--
Ryan (????)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://kirbyfan64.github.io/

On Dec 29, 2016 1:19 PM, "Shubha Ramani via pypy-dev" <pypy-dev at python.org>
wrote:

>
> What can I use then ? I need the address of the function ?
>
> [translation:ERROR] Exception: cannot use id() in RPython; see
> objectmodel.compute_xxx()
> Processing block:
>  block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>  in (pypy.module._vtune.interp_vtune:28)_get_full_name
>  containing the following operations:
>        buf_0 = simple_call((type RStringIO))
>        v943 = getattr(buf_0, ('write'))
>        v944 = getattr(pycode_0, ('co_code'))
>        v945 = simple_call(v943, v944)
>        v946 = getattr(buf_0, ('getsize'))
>        size_code_0 = simple_call(v946)
>        v947 = getattr(pycode_0, ('co_name'))
>        id_func_0 = id(v947)
>        v948 = getattr(pycode_0, ('co_name'))
>        v949 = simple_call((function _safe), v948)
>        v950 = getattr(pycode_0, ('co_firstlineno'))
>        v951 = getattr(pycode_0, ('co_filename'))
>        v952 = simple_call((function _safe), v951)
>        v953 = newtuple(v949, v950, v952, size_code_0, id_func_0)
>        v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953)
>  --end--
> [translation] start debugger...
> > /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()
> -> raise Exception("cannot use id() in RPython; "
> (Pdb+)
>
>
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161229/2393a7e9/attachment-0001.html>

From armin.rigo at gmail.com  Fri Dec 30 03:14:31 2016
From: armin.rigo at gmail.com (Armin Rigo)
Date: Fri, 30 Dec 2016 09:14:31 +0100
Subject: [pypy-dev] Debug build to analyze core dumps ?
In-Reply-To: <1735157935.3443706.1483047734009@mail.yahoo.com>
References: <807366974.3320388.1483031657635.ref@mail.yahoo.com>
 <807366974.3320388.1483031657635@mail.yahoo.com>
 <1735157935.3443706.1483047734009@mail.yahoo.com>
Message-ID: <CAMSv6X0Np0MUMmNAQ8pR4c27XcbRVkSzOCyzBQK4kqu_BiALBA@mail.gmail.com>

Hi Shubha,

On 29 December 2016 at 22:42, Shubha Ramani via pypy-dev
<pypy-dev at python.org> wrote:
> How do I build with debug symbols (i.e. -g) so that I can inspect core dumps
> when you get a traceback like this ?

Run "make clean; make lldebug" in the directory containing the
generated C sources (.../usession-xyz/testing_1).


A bient?t,

Armin.

From shubharamani at yahoo.com  Fri Dec 30 08:42:18 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 30 Dec 2016 05:42:18 -0800
Subject: [pypy-dev] cannot use "id" in Rpython
In-Reply-To: <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
 <1704861386.3369487.1483039114642@mail.yahoo.com>
 <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>
Message-ID: <1A8EE0A6-9316-45BF-BE10-66C80055B178@yahoo.com>

It's not a bad idea for me. I'm hooking up to a tool made by intel called "vtune" which requires the function name, function size and function address from the original python script code. I got the first two from Python reflection (the way VmProf does it) but I need the third (original script function  address). Is __repr__ supported in Rpython ? The string returned by __repr__ contains the Python address at least in Python. If id() doesn't work but __repr__ does then I can use it in the overloaded execute_frame and that should be fine.

Shubha 

> On Dec 29, 2016, at 4:29 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> 
> What exactly are you trying to do. Using id to get the address of a function is usually a bad idea...
> 
> --
> Ryan (????)
> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
> http://kirbyfan64.github.io/
> 
>> On Dec 29, 2016 1:19 PM, "Shubha Ramani via pypy-dev" <pypy-dev at python.org> wrote:
>> 
>> What can I use then ? I need the address of the function ?
>> 
>> [translation:ERROR] Exception: cannot use id() in RPython; see objectmodel.compute_xxx()
>> Processing block:
>>  block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'> 
>>  in (pypy.module._vtune.interp_vtune:28)_get_full_name 
>>  containing the following operations: 
>>        buf_0 = simple_call((type RStringIO)) 
>>        v943 = getattr(buf_0, ('write')) 
>>        v944 = getattr(pycode_0, ('co_code')) 
>>        v945 = simple_call(v943, v944) 
>>        v946 = getattr(buf_0, ('getsize')) 
>>        size_code_0 = simple_call(v946) 
>>        v947 = getattr(pycode_0, ('co_name')) 
>>        id_func_0 = id(v947) 
>>        v948 = getattr(pycode_0, ('co_name')) 
>>        v949 = simple_call((function _safe), v948) 
>>        v950 = getattr(pycode_0, ('co_firstlineno')) 
>>        v951 = getattr(pycode_0, ('co_filename')) 
>>        v952 = simple_call((function _safe), v951) 
>>        v953 = newtuple(v949, v950, v952, size_code_0, id_func_0) 
>>        v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953) 
>>  --end--
>> [translation] start debugger...
>> > /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()
>> -> raise Exception("cannot use id() in RPython; "
>> (Pdb+) 
>> 
>> 
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev
>> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161230/4075ead3/attachment.html>

From me at manueljacob.de  Fri Dec 30 11:04:32 2016
From: me at manueljacob.de (Manuel Jacob)
Date: Fri, 30 Dec 2016 17:04:32 +0100
Subject: [pypy-dev] cannot use "id" in Rpython
In-Reply-To: <1A8EE0A6-9316-45BF-BE10-66C80055B178@yahoo.com>
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
 <1704861386.3369487.1483039114642@mail.yahoo.com>
 <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>
 <1A8EE0A6-9316-45BF-BE10-66C80055B178@yahoo.com>
Message-ID: <b9ee8e713e108b9757c6033eecc91a5d@manueljacob.de>

The id of an object is not necessarily related in any way to the address 
of an object in Python.

In RPython, you can convert a function to a low-level function pointer 
with llhelper() from rpython.rtyper.annlowlevel.  You can call it like 
this:

fptr_type = lltype.Ptr(lltype.FuncType([], lltype.Void))
fptr = llhelper(fptr_type, func)

Depending on what you want to do with the function address exactly, you 
might need to cast the function pointer to a RPython "Address" with 
cast_ptr_to_adr() from rpython.rtyper.lltypesystem.llmemory.

On 2016-12-30 14:42, Shubha Ramani via pypy-dev wrote:
> It's not a bad idea for me. I'm hooking up to a tool made by intel
> called "vtune" which requires the function name, function size and
> function address from the original python script code. I got the first
> two from Python reflection (the way VmProf does it) but I need the
> third (original script function  address). Is __repr__ supported in
> Rpython ? The string returned by __repr__ contains the Python address
> at least in Python. If id() doesn't work but __repr__ does then I can
> use it in the overloaded execute_frame and that should be fine.
> 
> Shubha
> 
>> On Dec 29, 2016, at 4:29 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>> 
>> What exactly are you trying to do. Using id to get the address of a 
>> function is usually a bad idea...
>> 
>> --
>> Ryan (????)
>> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone 
>> else
>> http://kirbyfan64.github.io/
>> 
>>> On Dec 29, 2016 1:19 PM, "Shubha Ramani via pypy-dev" 
>>> <pypy-dev at python.org> wrote:
>>> 
>>> What can I use then ? I need the address of the function ?
>>> 
>>> [translation:ERROR] Exception: cannot use id() in RPython; see 
>>> objectmodel.compute_xxx()
>>> Processing block:
>>>  block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>>>  in (pypy.module._vtune.interp_vtune:28)_get_full_name
>>>  containing the following operations:
>>>        buf_0 = simple_call((type RStringIO))
>>>        v943 = getattr(buf_0, ('write'))
>>>        v944 = getattr(pycode_0, ('co_code'))
>>>        v945 = simple_call(v943, v944)
>>>        v946 = getattr(buf_0, ('getsize'))
>>>        size_code_0 = simple_call(v946)
>>>        v947 = getattr(pycode_0, ('co_name'))
>>>        id_func_0 = id(v947)
>>>        v948 = getattr(pycode_0, ('co_name'))
>>>        v949 = simple_call((function _safe), v948)
>>>        v950 = getattr(pycode_0, ('co_firstlineno'))
>>>        v951 = getattr(pycode_0, ('co_filename'))
>>>        v952 = simple_call((function _safe), v951)
>>>        v953 = newtuple(v949, v950, v952, size_code_0, id_func_0)
>>>        v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953)
>>>  --end--
>>> [translation] start debugger...
>>> > /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()
>>> -> raise Exception("cannot use id() in RPython; "
>>> (Pdb+)
>>> 
>>> 
>>> _______________________________________________
>>> pypy-dev mailing list
>>> pypy-dev at python.org
>>> https://mail.python.org/mailman/listinfo/pypy-dev
>>> 
> 
> _______________________________________________
> pypy-dev mailing list
> pypy-dev at python.org
> https://mail.python.org/mailman/listinfo/pypy-dev

From shubharamani at yahoo.com  Fri Dec 30 11:59:43 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 30 Dec 2016 08:59:43 -0800
Subject: [pypy-dev] cannot use "id" in Rpython
In-Reply-To: <b9ee8e713e108b9757c6033eecc91a5d@manueljacob.de>
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
 <1704861386.3369487.1483039114642@mail.yahoo.com>
 <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>
 <1A8EE0A6-9316-45BF-BE10-66C80055B178@yahoo.com>
 <b9ee8e713e108b9757c6033eecc91a5d@manueljacob.de>
Message-ID: <21737E83-B661-4961-9F9E-2A15B9D1F34F@yahoo.com>

This is very helpful. Thank you Manuel !
I will try your suggestion.

Shubha

> On Dec 30, 2016, at 8:04 AM, Manuel Jacob <me at manueljacob.de> wrote:
> 
> The id of an object is not necessarily related in any way to the address of an object in Python.
> 
> In RPython, you can convert a function to a low-level function pointer with llhelper() from rpython.rtyper.annlowlevel.  You can call it like this:
> 
> fptr_type = lltype.Ptr(lltype.FuncType([], lltype.Void))
> fptr = llhelper(fptr_type, func)
> 
> Depending on what you want to do with the function address exactly, you might need to cast the function pointer to a RPython "Address" with cast_ptr_to_adr() from rpython.rtyper.lltypesystem.llmemory.
> 
>> On 2016-12-30 14:42, Shubha Ramani via pypy-dev wrote:
>> It's not a bad idea for me. I'm hooking up to a tool made by intel
>> called "vtune" which requires the function name, function size and
>> function address from the original python script code. I got the first
>> two from Python reflection (the way VmProf does it) but I need the
>> third (original script function  address). Is __repr__ supported in
>> Rpython ? The string returned by __repr__ contains the Python address
>> at least in Python. If id() doesn't work but __repr__ does then I can
>> use it in the overloaded execute_frame and that should be fine.
>> Shubha
>>> On Dec 29, 2016, at 4:29 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>> What exactly are you trying to do. Using id to get the address of a function is usually a bad idea...
>>> --
>>> Ryan (????)
>>> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
>>> http://kirbyfan64.github.io/
>>>> On Dec 29, 2016 1:19 PM, "Shubha Ramani via pypy-dev" <pypy-dev at python.org> wrote:
>>>> What can I use then ? I need the address of the function ?
>>>> [translation:ERROR] Exception: cannot use id() in RPython; see objectmodel.compute_xxx()
>>>> Processing block:
>>>> block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>>>> in (pypy.module._vtune.interp_vtune:28)_get_full_name
>>>> containing the following operations:
>>>>       buf_0 = simple_call((type RStringIO))
>>>>       v943 = getattr(buf_0, ('write'))
>>>>       v944 = getattr(pycode_0, ('co_code'))
>>>>       v945 = simple_call(v943, v944)
>>>>       v946 = getattr(buf_0, ('getsize'))
>>>>       size_code_0 = simple_call(v946)
>>>>       v947 = getattr(pycode_0, ('co_name'))
>>>>       id_func_0 = id(v947)
>>>>       v948 = getattr(pycode_0, ('co_name'))
>>>>       v949 = simple_call((function _safe), v948)
>>>>       v950 = getattr(pycode_0, ('co_firstlineno'))
>>>>       v951 = getattr(pycode_0, ('co_filename'))
>>>>       v952 = simple_call((function _safe), v951)
>>>>       v953 = newtuple(v949, v950, v952, size_code_0, id_func_0)
>>>>       v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953)
>>>> --end--
>>>> [translation] start debugger...
>>>> > /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()
>>>> -> raise Exception("cannot use id() in RPython; "
>>>> (Pdb+)
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev


From shubharamani at yahoo.com  Fri Dec 30 13:10:49 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 30 Dec 2016 18:10:49 +0000 (UTC)
Subject: [pypy-dev] cannot use "id" in Rpython
In-Reply-To: <21737E83-B661-4961-9F9E-2A15B9D1F34F@yahoo.com>
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
 <1704861386.3369487.1483039114642@mail.yahoo.com>
 <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>
 <1A8EE0A6-9316-45BF-BE10-66C80055B178@yahoo.com>
 <b9ee8e713e108b9757c6033eecc91a5d@manueljacob.de>
 <21737E83-B661-4961-9F9E-2A15B9D1F34F@yahoo.com>
Message-ID: <1128804479.3847063.1483121449862@mail.yahoo.com>

I don't know the signature of the function coming into execute_frame ahead of time. Therefore as you suggested Manuel,I am leaving the argument is an empty list []. This doesn't seem to work though. See the assertion error I'm getting below.All other examples of this technique I see being used are passing in specific argument types to define a succinctfunction signature, i.e.?in test_warmstate.py:?ENTER_JIT = lltype.Ptr(lltype.FuncType([lltype.Signed, lltype.Float, ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? lltype.Signed], lltype.Bool)) ?

The following causes an assertion error:
?fun_name = _safe(pycode.co_name) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??fptr_type = lltype.Ptr(lltype.FuncType([], lltype.Void)) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??fptr = llhelper(fptr_type, fun_name) ? ??
[translation] start debugger...> /opt/shubha_vtune_pypy/rpython/rtyper/lltypesystem/lltype.py(776)constPtr()-> assert T.is_constant()
 

    On Friday, December 30, 2016 8:59 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 This is very helpful. Thank you Manuel !
I will try your suggestion.

Shubha

> On Dec 30, 2016, at 8:04 AM, Manuel Jacob <me at manueljacob.de> wrote:
> 
> The id of an object is not necessarily related in any way to the address of an object in Python.
> 
> In RPython, you can convert a function to a low-level function pointer with llhelper() from rpython.rtyper.annlowlevel.? You can call it like this:
> 
> fptr_type = lltype.Ptr(lltype.FuncType([], lltype.Void))
> fptr = llhelper(fptr_type, func)
> 
> Depending on what you want to do with the function address exactly, you might need to cast the function pointer to a RPython "Address" with cast_ptr_to_adr() from rpython.rtyper.lltypesystem.llmemory.
> 
>> On 2016-12-30 14:42, Shubha Ramani via pypy-dev wrote:
>> It's not a bad idea for me. I'm hooking up to a tool made by intel
>> called "vtune" which requires the function name, function size and
>> function address from the original python script code. I got the first
>> two from Python reflection (the way VmProf does it) but I need the
>> third (original script function? address). Is __repr__ supported in
>> Rpython ? The string returned by __repr__ contains the Python address
>> at least in Python. If id() doesn't work but __repr__ does then I can
>> use it in the overloaded execute_frame and that should be fine.
>> Shubha
>>> On Dec 29, 2016, at 4:29 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>> What exactly are you trying to do. Using id to get the address of a function is usually a bad idea...
>>> --
>>> Ryan (????)
>>> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
>>> http://kirbyfan64.github.io/
>>>> On Dec 29, 2016 1:19 PM, "Shubha Ramani via pypy-dev" <pypy-dev at python.org> wrote:
>>>> What can I use then ? I need the address of the function ?
>>>> [translation:ERROR] Exception: cannot use id() in RPython; see objectmodel.compute_xxx()
>>>> Processing block:
>>>> block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>>>> in (pypy.module._vtune.interp_vtune:28)_get_full_name
>>>> containing the following operations:
>>>>? ? ? buf_0 = simple_call((type RStringIO))
>>>>? ? ? v943 = getattr(buf_0, ('write'))
>>>>? ? ? v944 = getattr(pycode_0, ('co_code'))
>>>>? ? ? v945 = simple_call(v943, v944)
>>>>? ? ? v946 = getattr(buf_0, ('getsize'))
>>>>? ? ? size_code_0 = simple_call(v946)
>>>>? ? ? v947 = getattr(pycode_0, ('co_name'))
>>>>? ? ? id_func_0 = id(v947)
>>>>? ? ? v948 = getattr(pycode_0, ('co_name'))
>>>>? ? ? v949 = simple_call((function _safe), v948)
>>>>? ? ? v950 = getattr(pycode_0, ('co_firstlineno'))
>>>>? ? ? v951 = getattr(pycode_0, ('co_filename'))
>>>>? ? ? v952 = simple_call((function _safe), v951)
>>>>? ? ? v953 = newtuple(v949, v950, v952, size_code_0, id_func_0)
>>>>? ? ? v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953)
>>>> --end--
>>>> [translation] start debugger...
>>>> > /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()
>>>> -> raise Exception("cannot use id() in RPython; "
>>>> (Pdb+)
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161230/62d4af65/attachment.html>

From shubharamani at yahoo.com  Fri Dec 30 13:10:49 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Fri, 30 Dec 2016 18:10:49 +0000 (UTC)
Subject: [pypy-dev] cannot use "id" in Rpython
In-Reply-To: <21737E83-B661-4961-9F9E-2A15B9D1F34F@yahoo.com>
References: <1704861386.3369487.1483039114642.ref@mail.yahoo.com>
 <1704861386.3369487.1483039114642@mail.yahoo.com>
 <CAO41-mMhp0QQtS+ZdckAQhf0sOH6Q5gqKMGWxippiJA2ttuytw@mail.gmail.com>
 <1A8EE0A6-9316-45BF-BE10-66C80055B178@yahoo.com>
 <b9ee8e713e108b9757c6033eecc91a5d@manueljacob.de>
 <21737E83-B661-4961-9F9E-2A15B9D1F34F@yahoo.com>
Message-ID: <1128804479.3847063.1483121449862@mail.yahoo.com>

I don't know the signature of the function coming into execute_frame ahead of time. Therefore as you suggested Manuel,I am leaving the argument is an empty list []. This doesn't seem to work though. See the assertion error I'm getting below.All other examples of this technique I see being used are passing in specific argument types to define a succinctfunction signature, i.e.?in test_warmstate.py:?ENTER_JIT = lltype.Ptr(lltype.FuncType([lltype.Signed, lltype.Float, ? ? ? ? ? ? ? ? ? ?? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? lltype.Signed], lltype.Bool)) ?

The following causes an assertion error:
?fun_name = _safe(pycode.co_name) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??fptr_type = lltype.Ptr(lltype.FuncType([], lltype.Void)) ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??fptr = llhelper(fptr_type, fun_name) ? ??
[translation] start debugger...> /opt/shubha_vtune_pypy/rpython/rtyper/lltypesystem/lltype.py(776)constPtr()-> assert T.is_constant()
 

    On Friday, December 30, 2016 8:59 AM, Shubha Ramani <shubharamani at yahoo.com> wrote:
 

 This is very helpful. Thank you Manuel !
I will try your suggestion.

Shubha

> On Dec 30, 2016, at 8:04 AM, Manuel Jacob <me at manueljacob.de> wrote:
> 
> The id of an object is not necessarily related in any way to the address of an object in Python.
> 
> In RPython, you can convert a function to a low-level function pointer with llhelper() from rpython.rtyper.annlowlevel.? You can call it like this:
> 
> fptr_type = lltype.Ptr(lltype.FuncType([], lltype.Void))
> fptr = llhelper(fptr_type, func)
> 
> Depending on what you want to do with the function address exactly, you might need to cast the function pointer to a RPython "Address" with cast_ptr_to_adr() from rpython.rtyper.lltypesystem.llmemory.
> 
>> On 2016-12-30 14:42, Shubha Ramani via pypy-dev wrote:
>> It's not a bad idea for me. I'm hooking up to a tool made by intel
>> called "vtune" which requires the function name, function size and
>> function address from the original python script code. I got the first
>> two from Python reflection (the way VmProf does it) but I need the
>> third (original script function? address). Is __repr__ supported in
>> Rpython ? The string returned by __repr__ contains the Python address
>> at least in Python. If id() doesn't work but __repr__ does then I can
>> use it in the overloaded execute_frame and that should be fine.
>> Shubha
>>> On Dec 29, 2016, at 4:29 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
>>> What exactly are you trying to do. Using id to get the address of a function is usually a bad idea...
>>> --
>>> Ryan (????)
>>> Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
>>> http://kirbyfan64.github.io/
>>>> On Dec 29, 2016 1:19 PM, "Shubha Ramani via pypy-dev" <pypy-dev at python.org> wrote:
>>>> What can I use then ? I need the address of the function ?
>>>> [translation:ERROR] Exception: cannot use id() in RPython; see objectmodel.compute_xxx()
>>>> Processing block:
>>>> block at 19 is a <class 'rpython.flowspace.flowcontext.SpamBlock'>
>>>> in (pypy.module._vtune.interp_vtune:28)_get_full_name
>>>> containing the following operations:
>>>>? ? ? buf_0 = simple_call((type RStringIO))
>>>>? ? ? v943 = getattr(buf_0, ('write'))
>>>>? ? ? v944 = getattr(pycode_0, ('co_code'))
>>>>? ? ? v945 = simple_call(v943, v944)
>>>>? ? ? v946 = getattr(buf_0, ('getsize'))
>>>>? ? ? size_code_0 = simple_call(v946)
>>>>? ? ? v947 = getattr(pycode_0, ('co_name'))
>>>>? ? ? id_func_0 = id(v947)
>>>>? ? ? v948 = getattr(pycode_0, ('co_name'))
>>>>? ? ? v949 = simple_call((function _safe), v948)
>>>>? ? ? v950 = getattr(pycode_0, ('co_firstlineno'))
>>>>? ? ? v951 = getattr(pycode_0, ('co_filename'))
>>>>? ? ? v952 = simple_call((function _safe), v951)
>>>>? ? ? v953 = newtuple(v949, v950, v952, size_code_0, id_func_0)
>>>>? ? ? v954 = mod(('vtune:%s:%d:%s:%d:%d'), v953)
>>>> --end--
>>>> [translation] start debugger...
>>>> > /opt/shubha_vtune_pypy/rpython/annotator/unaryop.py(188)id()
>>>> -> raise Exception("cannot use id() in RPython; "
>>>> (Pdb+)
>>>> _______________________________________________
>>>> pypy-dev mailing list
>>>> pypy-dev at python.org
>>>> https://mail.python.org/mailman/listinfo/pypy-dev
>> _______________________________________________
>> pypy-dev mailing list
>> pypy-dev at python.org
>> https://mail.python.org/mailman/listinfo/pypy-dev

   
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161230/62d4af65/attachment-0003.html>

From vierkantor at vierkantor.com  Sat Dec 31 10:30:27 2016
From: vierkantor at vierkantor.com (Tim Baanen)
Date: Sat, 31 Dec 2016 15:30:27 +0000
Subject: [pypy-dev] AssertionError in parse_ebnf when nesting + and []
Message-ID: <CAGeL8G9J9WdbSAv5OjXrYWO2A4-J7JYFKU=eQPfqAid0JRWuKg@mail.gmail.com>

Hi everyone,

This is a copy of an issue I filed at your issue tracker,
https://bitbucket.org/pypy/pypy/issues/2459/assertionerror-in-parse_ebnf-
when-nesting, where I was told to report it to the mailing list. I was
using the rlib.parsing module for parsing a simple language, but I ran into
an AssertionError which I can't explain (from the documentation, at least).
It seems to occur whenever a symbol like `+` or `*` is used inside `[]`
brackets. Here is an example file:

from rpython.rlib.parsing.ebnfparse import parse_ebnf
> regexes, rules, ToAST = parse_ebnf("""
> test: ["foo"+] "bar";
> """)
>

When I run this code, both on the default branch and on the
`release-pypy2.7-v5.6.0` branch, the following error occurs:

test/test_parser.py:10: in <module>
>     """)
> pypy/rpython/rlib/parsing/ebnfparse.py:57: in parse_ebnf
>     s.visit(visitor)
> pypy/rpython/rlib/parsing/tree.py:86: in visit
>     return getattr(visitor, "visit_" + self.symbol)(self)
> pypy/rpython/rlib/parsing/ebnfparse.py:110: in visit_file
>     return node.children[0].visit(self)
> pypy/rpython/rlib/parsing/tree.py:86: in visit
>     return getattr(visitor, "visit_" + self.symbol)(self)
> pypy/rpython/rlib/parsing/ebnfparse.py:114: in visit_list
>     child.visit(self)
> pypy/rpython/rlib/parsing/tree.py:86: in visit
>     return getattr(visitor, "visit_" + self.symbol)(self)
> pypy/rpython/rlib/parsing/ebnfparse.py:131: in visit_production
>     expansions = node.children[2].visit(self)
> pypy/rpython/rlib/parsing/tree.py:86: in visit
>     return getattr(visitor, "visit_" + self.symbol)(self)
> pypy/rpython/rlib/parsing/ebnfparse.py:146: in visit_body
>     expansion = child.visit(self)
> pypy/rpython/rlib/parsing/tree.py:86: in visit
>     return getattr(visitor, "visit_" + self.symbol)(self)
> pypy/rpython/rlib/parsing/ebnfparse.py:153: in visit_expansion
>     expansion = child.visit(self)
> pypy/rpython/rlib/parsing/tree.py:86: in visit
>     return getattr(visitor, "visit_" + self.symbol)(self)
> pypy/rpython/rlib/parsing/ebnfparse.py:161: in visit_enclosed
>     assert change == " " or change == newchange
> AssertionError
>

Commenting out the assertion that is failing doesn't seem to cause any
problems and the resulting parser has the right behavior. My current
workaround is to make a new terminal symbol that behaves like `"foo"+` in
the example code, which appears to work but is somewhat less readable.

Did I miss something or is this an actual bug in RPython and/or its
documentation?

Thanks in advance!
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161231/761e3397/attachment.html>

From shubharamani at yahoo.com  Sat Dec 31 18:56:47 2016
From: shubharamani at yahoo.com (Shubha Ramani)
Date: Sat, 31 Dec 2016 23:56:47 +0000 (UTC)
Subject: [pypy-dev] [translation:ERROR] AnnotatorError: when I try to pass a
 variable down to x86/assembler.py
References: <1434469698.4276065.1483228607376.ref@mail.yahoo.com>
Message-ID: <1434469698.4276065.1483228607376@mail.yahoo.com>

Trying to pass a dummy variable called "shubha" to the X86 Assembler.?Made the following changes to code, am getting the following errors. Please help:rpython/jit/backend/llsupport/llmodel.py

class AbstractLLCPU(AbstractCPU):?def __init__(self, rtyper, stats, opts, shubha, translate_support_code=Fa\
lse, ?gcdescr=None):

?self.setup(self.shubha)?

/rpython/jit/backend/llsupport/assembler.py
class BaseAssembler(object): ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? """ Base class for Assembler generator in real backends ? ? ? ? ? ? ? ? ??? ? """ ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? def __init__(self, cpu, shubha, translate_support_code=False):?
/rpython/jit/backend/x86/assembler.py

class Assembler386(BaseAssembler, VectorAssemblerMixin): ? ? ? ? ? ? ? ? ? ? ?? ? _regalloc = None ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?? ? _output_loop_log = None ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? _second_tmp_reg = ecx ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? DEBUG_FRAME_DEPTH = False ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ??? ? def __init__(self, cpu, shubha, translate_support_code=False): ? ? ? ? ? ?? ? ? ? BaseAssembler.__init__(self, cpu, shubha, translate_support_code)


..........+**%%%%%%**++++++++++++++++++++++++%[13772] translation-task}?[Timer] Timings:[Timer] annotate ? ? ? ? ? ? ? ? ? ? ? --- 301.5 s[Timer] rtype_lltype ? ? ? ? ? ? ? ? ? --- 315.2 s[Timer] pyjitpl_lltype ? ? ? ? ? ? ? ? --- 177.8 s[Timer] ==========================================[Timer] Total: ? ? ? ? ? ? ? ? ? ? ? ? --- 794.5 s[translation:info] Error:? ?File "/opt/shubha_vtune_pypy/rpython/translator/goal/translate.py", line 317, in main? ? drv.proceed(goals)? ?File "/opt/shubha_vtune_pypy/rpython/translator/driver.py", line 551, in proceed? ? result = self._execute(goals, task_skip = self._maybe_skip())? ?File "/opt/shubha_vtune_pypy/rpython/translator/tool/taskengine.py", line 114, in _execute? ? res = self._do(goal, taskcallable, *args, **kwds)? ?File "/opt/shubha_vtune_pypy/rpython/translator/driver.py", line 278, in _do? ? res = func()? ?File "/opt/shubha_vtune_pypy/rpython/translator/driver.py", line 361, in task_pyjitpl_lltype? ? backend_name=self.config.translation.jit_backend, inline=True)? ?File "/opt/shubha_vtune_pypy/rpython/jit/metainterp/warmspot.py", line 64, in apply_jit? ? warmrunnerdesc.finish()? ?File "/opt/shubha_vtune_pypy/rpython/jit/metainterp/warmspot.py", line 281, in finish? ? self.annhelper.finish()? ?File "/opt/shubha_vtune_pypy/rpython/rtyper/annlowlevel.py", line 217, in finish? ? self.finish_annotate()? ?File "/opt/shubha_vtune_pypy/rpython/rtyper/annlowlevel.py", line 236, in finish_annotate? ? ann.complete_helpers(self.policy)? ?File "/opt/shubha_vtune_pypy/rpython/annotator/annrpython.py", line 116, in complete_helpers? ? self.complete()? ?File "/opt/shubha_vtune_pypy/rpython/annotator/annrpython.py", line 208, in complete? ? raise annmodel.AnnotatorError(text)[translation:ERROR] AnnotatorError:??Blocked block -- operation cannot succeed?? ? v3556 = getattr(self_279, ('loc_m'))?In <FunctionGraph of (rpython.jit.backend.x86.regloc:251)AddressLoc.find_unused_reg at 0x18ca1da8>:Happened at file /opt/shubha_vtune_pypy/rpython/jit/backend/x86/regloc.py line 253?==> ? ? ? ? ? ? if self.loc_m[0] == eax.value:?Known variable annotations:?self_279 = SomeInstance(can_be_None=False, classdef=rpython.jit.backend.x86.regloc.AddressLoc)?Blocked block -- operation cannot succeed?? ? v3557 = getattr(self_280, ('loc_a'))?In <FunctionGraph of (rpython.jit.backend.x86.regloc:251)AddressLoc.find_unused_reg at 0x18ca1da8>:Happened at file /opt/shubha_vtune_pypy/rpython/jit/backend/x86/regloc.py line 256?==> ? ? ? ? ? ? if self.loc_a[0] == eax.value:?Known variable annotations:?self_280 = SomeInstance(can_be_None=False, classdef=rpython.jit.backend.x86.regloc.AddressLoc)?Blocked block -- operation cannot succeed?? ? v3558 = getattr(self_281, ('assembler'))?In <FunctionGraph of (rpython.jit.backend.x86.regalloc:160)RegAlloc._prepare at 0x87b47130>:Happened at file /opt/shubha_vtune_pypy/rpython/jit/backend/x86/regalloc.py line 163?==> ? ? ? ? cpu = self.assembler.cpu? ? ? ? ? ? self.fm = X86FrameManager(cpu.get_baseofs_of_frame_field())? ? ? ? ? ? operations = cpu.gc_ll_descr.rewrite_assembler(cpu, operations,? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?allgcrefs)? ? ? ? ? ? # compute longevity of variables? ? ? ? ? ? longevity, last_real_usage = compute_vars_longevity(? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? inputargs, operations)?Known variable annotations:?self_281 = SomeInstance(can_be_None=False, classdef=rpython.jit.backend.x86.regalloc.RegAlloc)?Blocked block -- operation cannot succeed?? ? v3559 = getattr(self_282, ('loc_m'))?In <FunctionGraph of (rpython.jit.backend.x86.regloc:248)AddressLoc.value_m at 0x9a43bb0>:Happened at file /opt/shubha_vtune_pypy/rpython/jit/backend/x86/regloc.py line 249?==> ? ? ? ? return self.loc_m?Known variable annotations:?self_282 = SomeInstance(can_be_None=False, classdef=rpython.jit.backend.x86.regloc.AddressLoc)?Blocked block -- operation cannot succeed?? ? v3560 = getattr(self_283, ('loc_a'))?In <FunctionGraph of (rpython.jit.backend.x86.regloc:245)AddressLoc.value_a at 0x76b0bb8>:Happened at file /opt/shubha_vtune_pypy/rpython/jit/backend/x86/regloc.py line 246?==> ? ? ? ? return self.loc_a?Known variable annotations:?self_283 = SomeInstance(can_be_None=False, classdef=rpython.jit.backend.x86.regloc.AddressLoc)[translation] start debugger...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/pypy-dev/attachments/20161231/eb4cecf6/attachment-0001.html>