From ruos at ofle.com Sun Feb 2 10:17:23 2014 From: ruos at ofle.com (=?GB2312?B?x+vXqs/gudiyv8PF?=) Date: Sun, 2 Feb 2014 17:17:23 +0800 Subject: [pypy-dev] =?gb2312?b?ytCzocf9tq+1xNDCsvrGt7+qt6LB97PMus253MDt?= =?gb2312?b?cHlweS1kZXY=?= Message-ID: <3fH6Sp1M5pz7Ljb@mail.python.org> ??????????????????? pypy-dev at codespeak.net 2014?3?27-28??? 2014?3?24-25??? 2014?3?20-21??? 2014?6?30-7.1??? 2014?6?23-24??? 2014?6?26-27??? 2014?10?30-31??? 2014?10?27-28 ??? 2014?10?23-24??? ?????4300?/?? *????,????? ,??????2800???????????????????? ????????????????????????????? ???????CEO/?????????/???????/???????????/????? PMO???????????????????????? ?????021-51602856 0755-61287552 020-80560638 ??????????????tuixin2013 at 126.com??? =============================================================================== ?????????/????????????????????????????? 1.?????????????????????????????? 2.????????????????????????????????????????? 3.???????????????????????????????????? 4.???????????????????????????????????????????????????????? 5.???????????????????????????????????????????????????????? ????????????????????????????? 6.???????????????????? 7.????????????????????????????? 8.?????????????????????????????????? ????????????????????????????????????????????????????? ???????????????????????????????????????????????????? ????????????? ----------------------------------------------------------------------- 1.????300???????????????????????????????????????? 2.???????????????????????????????????????? 3.??????????????????????????????????? 4.??????????????????????? 5.??????????????????????????????? 6.????????50??????????????????????????????????Action Plan??? ??????????????????????????? -------------------------------------------------------------------------------------------------- ?????? ?????????? 1???????????????????????????IT???? 2???????????? 3???????????????NPD?CMMI?ISO9000???? 4???????????????? 5???????? 6???????????????????????????????? 7???????????????? 8?????????????????? 9??????????????? 10???????? ???????????? 1?????????????? 2?????????? a)????? b)????? c)????? 3????????????????? 4?????????????????? a)??????????????????????????????? b)????????????????????? c)????????????????? d)????????????????? e)??????????????????????? 5???????????????????? a)?????????? b)???? c)????????????? d)???????????? 6??????????????? 7????????????????????? 8????????????????????????? 9??????????????/????????? 10???????? ???????????????????????????? 1?????????????????? a)?????????? b)???????????? c)??????????????????????? d)?????????????? e)?????????????? 2????????????? 3?????????????????????????? 4??????????????????????????????????? 5????????????????? 6?????????????? 7???????????? 8???????????? 9?????????????? 10??????????????? 11?????????PMO? 12?????????????? 13????????????????????? 14???????? ???????????? 1?????????????? 2????????????????? 3???????????? a??????????? b??????????????????? c??????????????????? d?????????????????? e??????????????????????????? 4???????????????????? a) ?????? b)?????? c)??????????? d)??????????????? 5??????????????????????? ?????????????????? 6????????????????????????????? 7???????? ?????????? 1???????? a) ?????? b) ?????? 2???????? a) ????????????????????? b)????????? c)??????????? d)????????? e)????????? f)??????????????? g)?????????????? h)WBS????????? i)WBS??????? j)PBS?WBS?OBS?RBS??????? k)????????? l)??????????? m)PERT???? n)?????????? ?????? ?????? o)??????? 3????????????????WBS????????? ?????????? 1??????????????????? 2????????????? 3????? a??????? b???????? c???????? 4?????????????? a) ?????? b) ?????? 5?????????????? a) ?????? b) ??????? 6???????????????? a) ?????? b) ?????? 7?????????????? 8???????????????? 9????????????????? 10?????????????? 11???????????????? 12???????????????? a) ????? b) ????? 13?????????????? 14?????????????? 15?????????????? a) ???? b) ???????? c) ???? 16??????????QA???? 17?????????????? a) ????? b) ??????? c) ???? 18?????????????? 19?????????????? a) ???????? b) ???? c) ???????? 20???????? ??????????????????? 1???????????????????????????????????????? 2???????????????????????? 3???????????? 4???????????? 5??????????? 6?????????????? 7?????????????? --------------------------------------------- ?? ??????????????????????????????????????????????? ?????????????????????????????????????????????? ?????????????????????????????????????????????? ?????????????????????????????????????????????? ?????????????BFL?Infosys??????????????CMM???????????? ??????????????????? ??????????????????????10??????????????????????? ???????????????????????????????????? From matheus2740 at gmail.com Mon Feb 3 03:55:41 2014 From: matheus2740 at gmail.com (Matheus Salvia) Date: Mon, 3 Feb 2014 00:55:41 -0200 Subject: [pypy-dev] Help with memory usage Message-ID: Sorry for mailing this list just to ask, but I have a question regarding pypy's memory usage. I have an application which starts many child processes in order to do heavy processing. When running my app under CPython, it uses about 1GB of memory, but when running with pypy it goes up to almost 3GB. I tried calling gc.collect() at some points on the code as well as setting some GC ambient vars, but without much success. Why is pypy using so much more memory, and how can I reduce it? Is there a way of using refcount instead of GC? Thanks in Advance, Matheus Salvia -------------- next part -------------- An HTML attachment was scrubbed... URL: From anto.cuni at gmail.com Mon Feb 3 14:27:54 2014 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 3 Feb 2014 14:27:54 +0100 Subject: [pypy-dev] Europython 2014 Message-ID: Hi Armin, hi Romain, in Leysin we discussed about the idea of giving the usual "PyPy status talk" at the upcoming EuroPython, considering that people complained that there was none at the last one :) The CfP ends on the 9th: should we file a joint proposal? Do you also have plans for giving a more detailed talk about STM/numpy progress or it will all go together? Is there anyone else coming to EP interested in doing it? ciao, Anto -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Mon Feb 3 19:27:26 2014 From: arigo at tunes.org (Armin Rigo) Date: Mon, 3 Feb 2014 19:27:26 +0100 Subject: [pypy-dev] Help with memory usage In-Reply-To: References: Message-ID: Hi Matheus, On 3 February 2014 03:55, Matheus Salvia wrote: > When running my app under CPython, it uses about 1GB of memory, but when > running with pypy it goes up to almost 3GB. This is a question that doesn't have a single answer. You need to give us a lot more information about what your application does, before we can start proposing ideas about why there is such a difference in your case. In our own experience, the memory usage is usually very roughly around the same as CPython (a factor less than 2 in one way or another). As a first (likely completely random) guess, you are starting a very large number of processes, each of which consumes 60MB instead of 20MB in CPython. If I'm correct, then there is not much we can do; it's known that PyPy's baseline memory usage is higher than CPython's, due to the JIT which ends up storing a few dozens of MBs in any process. A bient?t, Armin. From matheus2740 at gmail.com Mon Feb 3 20:07:29 2014 From: matheus2740 at gmail.com (Matheus Salvia) Date: Mon, 3 Feb 2014 17:07:29 -0200 Subject: [pypy-dev] Help with memory usage In-Reply-To: References: Message-ID: Thanks for the answer, Armin. The application is corrently using about 15 processes, but this is a configurable number. By your information I can assume that the process overhead would be of about 600MB, which is not a negligible number but still doesn't fully explain all of the memory usage. The application is an information processor, and simplifying the algorithm it would be something like: - start a worker process pool - receive messages from a queue - delegate messages to the workers The workers themselves have a lifecycle about like that: - get message - fetch data pointed by the message from database (apache cassandra) - process data: - - call a C library to do some preprocessing (Zorba XQuery thru CFFI) - - cal a python module to do intermediate and post processing - store resulting data to database (ElasticSearch) The libraries I'm using are: pycassa, pyelasticsearch, cffi, boto and peewee My operating system is Ubuntu 12.04 LTS x64 Last but not least, is there a way to change pypy from using a GC to using refcount like CPython? That was the information you needed? Feel free to ask anything. Again, thank you, Matheus Salvia 2014-02-03 Armin Rigo : > Hi Matheus, > > On 3 February 2014 03:55, Matheus Salvia wrote: > > When running my app under CPython, it uses about 1GB of memory, but when > > running with pypy it goes up to almost 3GB. > > This is a question that doesn't have a single answer. You need to > give us a lot more information about what your application does, > before we can start proposing ideas about why there is such a > difference in your case. In our own experience, the memory usage is > usually very roughly around the same as CPython (a factor less than 2 > in one way or another). > > As a first (likely completely random) guess, you are starting a very > large number of processes, each of which consumes 60MB instead of 20MB > in CPython. If I'm correct, then there is not much we can do; it's > known that PyPy's baseline memory usage is higher than CPython's, due > to the JIT which ends up storing a few dozens of MBs in any process. > > > A bient?t, > > Armin. > -- -- // Matheus Salvia Desenvolvedor Mobile Celular: +55 11 9-6446-2332 Skype: meta.faraday -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Mon Feb 3 20:46:36 2014 From: arigo at tunes.org (Armin Rigo) Date: Mon, 3 Feb 2014 20:46:36 +0100 Subject: [pypy-dev] Help with memory usage In-Reply-To: References: Message-ID: Hi Matheus, On 3 February 2014 20:07, Matheus Salvia wrote: > Last but not least, is there a way to change pypy from using a GC to using > refcount like CPython? No, PyPy cannot use refcounting. > That was the information you needed? This is still too vague to know why memory usage is so high. I think that you'll have to post it to the bug tracker for us to investigate. You can post the whole application; it is fine as long as we can reproduce the problem. Of course, if your application is closed source, then this method doesn't work; in this case we can discuss commercial support solutions. Also note that you said it's using 3GB instead of 1GB, which divided by 15 processes makes 200MB instead of 66MB. Such numbers are at the extreme range of "expected": if each process runs for a long time, it can potentially accumulate 50-100MB of JITted code in it. A bient?t, Armin. From kennylevinsen at gmail.com Mon Feb 3 20:45:35 2014 From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen) Date: Mon, 3 Feb 2014 20:45:35 +0100 Subject: [pypy-dev] Help with memory usage In-Reply-To: References: Message-ID: <75D0178D-B53B-4C3C-BE8B-7344AA0021E3@gmail.com> Hi, Sorry for intervening, but a quick answer to your last request: No. There are multiple pluggable GC?s that PyPy can compile with, incminimark being the default now IIRC, but refcounting is not an option. It is unlikely to do you any good, though. It saves you a bit of memory between the apparent death of an object and its collection, but that shouldn?t be that bad. With the new incminimark, that should have improved a lot. Can you try to reduce the memory leak to a reproducible example? That would be terribly helpful. Unless there is a known issue with one of those libraries, there isn?t enough information for the issue to be debugged. Best regards, Kenny Levinsen On 03 Feb 2014, at 20:07, Matheus Salvia wrote: > Thanks for the answer, Armin. > The application is corrently using about 15 processes, but this is a configurable number. > By your information I can assume that the process overhead would be of about 600MB, > which is not a negligible number but still doesn't fully explain all of the memory usage. > The application is an information processor, and simplifying the algorithm it would be something like: > - start a worker process pool > - receive messages from a queue > - delegate messages to the workers > > The workers themselves have a lifecycle about like that: > - get message > - fetch data pointed by the message from database (apache cassandra) > - process data: > - - call a C library to do some preprocessing (Zorba XQuery thru CFFI) > - - cal a python module to do intermediate and post processing > - store resulting data to database (ElasticSearch) > > The libraries I'm using are: pycassa, pyelasticsearch, cffi, boto and peewee > My operating system is Ubuntu 12.04 LTS x64 > > Last but not least, is there a way to change pypy from using a GC to using refcount like CPython? > > That was the information you needed? > Feel free to ask anything. > > Again, thank you, > Matheus Salvia > > > 2014-02-03 Armin Rigo : > Hi Matheus, > > On 3 February 2014 03:55, Matheus Salvia wrote: > > When running my app under CPython, it uses about 1GB of memory, but when > > running with pypy it goes up to almost 3GB. > > This is a question that doesn't have a single answer. You need to > give us a lot more information about what your application does, > before we can start proposing ideas about why there is such a > difference in your case. In our own experience, the memory usage is > usually very roughly around the same as CPython (a factor less than 2 > in one way or another). > > As a first (likely completely random) guess, you are starting a very > large number of processes, each of which consumes 60MB instead of 20MB > in CPython. If I'm correct, then there is not much we can do; it's > known that PyPy's baseline memory usage is higher than CPython's, due > to the JIT which ends up storing a few dozens of MBs in any process. > > > A bient?t, > > Armin. > > > > -- > -- > // Matheus Salvia > Desenvolvedor Mobile > Celular: +55 11 9-6446-2332 > Skype: meta.faraday > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From chartering at potephos.com Mon Feb 3 21:29:33 2014 From: chartering at potephos.com (Potephos Shipping and Trading Limited/Chartering Department) Date: Mon, 3 Feb 2014 22:29:33 +0200 Subject: [pypy-dev] Looking for Capsize of 150.000 ton Message-ID: Good day, dear Sirs! please assist with the following: 1. Service : Maritime freight between Venezuela and China. 2. Ship : Capsize of 150.000 ton 3. Mineral : Iron Ore 4. Port of Origin : VENEZUELA - Puerto BOCA GRANDE ( TOLL - The ship must be moved 2 times to finish loading in Boca Grande) 5. Destination : China - Qindao Port 6. Monthly shipments : 1 7. Duration : Spot (1? Shipment) 8. First Shipment : 09 / 01 til 1/15/14 . 9. Payment : 30% TT, upon signing the contract 70% + 100% against LC BL. 10. Paying Bank : 1? Koera Exchange Bank na Japon - Tokio 2? Shinhan Bank no Japon - Tokio 11. Price target: USD 25.50 12. Immediate signing of contract : 48 hours 13. Loading terms: 7 days 14. Loading times : 5 days ( 30,000 ton / day ) maximum. 15. Quantity to be discharged in discharge port: Total discharge in Qindao 16. Terms Discharge Time: 8 days 17. Discharge times : 6 days ( 25,000 ton / day ) maximum. 18. Cargo Loaded : 5 days ( 30,000 ton / day ) 19. Demurrage / Dispatch terms: $ 28.00 ( Ferrominera Orinoco) 20. Total comission: 3.75 % on F/D/D -Normal Stowage Factor of Bulk Iron Ore Ferrominera Orinoco (Shippers) use the following Stowage factor for their iron ore shipments. ? Iron ore fines : 0,45 - 0,55 m3/MT (dry) ? Sinter feed: 0,4 -0,5 m3/MT (dry) -Port Restriction (Boca Grande II Transfer Station) Vessels nominated to load at Boca Grande II Transfer Station shall have clear and unobstructed holds and shall be suitable for loading iron ore. ? Maximum vessel height from waterline to top, or railing at vessel side, or top of hatch coamings: 17,20 meters. ? Maximum distance between centers of extreme holds: 216,00 meters. ? Maximum beam: 55,00 meters, if distance between centers of extreme hatches is less than 187,00 meters and 52,00 m, if distance is greater, up to 216,00 meters. Exceptions to these limitations to be considered on a case by case basis. Boca Grande II Transfer Station Specifics ? Loa 314 M ? Lbp 300 M ? Breadth 50 M ? Draught 20,41 M ? Cargo Capacity (Aprox) 85,000 MT ? Unloading Rate (Design) 6000 Tph ? Unloading Rate (Average) 3500 Tph ? 4000 Tph ? Max air draft 17,2 M -Normal Loading rate of Bulk iron ore Unloading rate (Average) 3500 Tph ? 4000 Tph Transfer station works Monday ? Sunday SHINC 24 hours in 3 shifts. 07:00-15:00, 15:00-23:00, 23:00-07:00, if wp. -Loading method The transfer station employs gravity fed belt discharging system with final transfer cargo to the ocean vessel being via a traveling ship-loader with a 36m conveyor boom. The boom can be slewed outboard 100 degrees from the centerline and the ship-loader travels along the port side is 187m , eliminating the need for any shifting of the ocean vessel while loading. No shifting alongside will be required during loading provided initial positioning during berthing is done correctly. -Normal waiting time to be alongside (Congestion) Waiting time depends on vessel line up at transfer station and availability of material at station. At this time there is only 1 vessel loading at the station. -Enroute time from pilot station to berth or anchorage Your good vessel will arrive Boca Grande II transfer station (BGIITS) and anchor while waiting for instructions from station in order to berth. Once vessel is given berthing instructions docking pilot will be picked up at station via helicopter and board your good vessel. Docking maneuvers depending on weather should take approximately 2 hours agw, wp. -Weather condition January/Febuary Venezuela is located in a tropical region. At this time we are ongoing our dry season. However it is not uncommon to get random showers during our dry season. Dry season will see continue until around the months of April/May -How many days taken by last agency?s last vessel vessel Agency Last vessel to load at Boca Grande II Transfer Station took 7 days to complete loading operations and set sail to destination. She loaded approx. 150.00MT General information of Boca Grande II Transfer Station. Location: (LAT 90?56?04?N LONG 61?34?48?W) in Venezuelan territorial waters in Serpent's Mouth between Trinidad and delta of the Orinoco river. The transfer station is fixed by a three legged permanent mooring system. To avoid interference with the mooring system, the export vessel shall not use her anchors in the area of a circle of 1,000 meters of diameter with center located at the bow of the transfer station. Export vessels nominated shall have clear and unobstructed holds and shall be suitable for loading iron ore. Each vessel must be clean to receive iron ore on presentation at the transfer station for loading. Allowable dimensions as we mentioned previously are as follows: ? Maximum vessel height from waterline to top, or railing at vessel side, or top of hatch coamings: 17,20 meters. ? Maximum distance between centers of extreme holds: 216,00 meters. ? Maximum beam: 55,00 meters, if distance between centers of extreme hatches is less than 187,00 meters and 52,00 m, if distance is greater, up to 216,00 meters. Exceptions to these limitations to be considered on a case by case basis. There are 2 shuttles that transfer material to TSBGII from Ferrominera Orinoco. ? Mv Rio Caroni ? Mv General Piar. On average the each shuttle can transfer, ? Dry Season (October ? May) ??. 40.000-50.000 MT (approx.) ? Rainy Season (May to September) ?.. 50.000-60.000 MT (approx.) Masters of vessels accepted to load at the transfer station shall give 72, 24, 12 and 6 hrs notice of E.T.A. To the transfer station via agent. Master shall send their cargo loading sequence via agent at least 24 hrs before E.T.A, who will pass a copy to master "Boca Grande". When the vessel is in all respects ready to load, a notice of readiness must be given at any time of day or night to the station, via vhf ch.16. Should the masters fail to give the above notices of the vessel's E.T.A. At the transfer station, they may be kept waiting for the same period as that of the longest notice required here and not given. Vessels are advised to provide themselves with navigation charts No. USHO (DMA) 24390 and 196, and BA 481. Upon arrival at the transfer station area, vessels shall, unless otherwise instructed, anchor with at least 8 shots of chain in the water, in the vicinity of the station, anchorages areas suggested are between: 09-56-40 AND 09-57-30 N, 61-37-40 AND 61-38-45 W. OR BETWEEN: 09-55-50 AND 09-57-30 N, 61-32-40 AND 61-34-40 W. The ship will agree with the station on a vhf working channel for communication during loading, preferably ch.9. Masters shall take all the necessary measures to avoid by all means discharge or spillage of fuel or dirty ballast in the area. Masters shall assure dispatch of vessels by De-ballasting the vessels consistent with safety before arriving to load at the transfer station, consistent however with maximum vessel's air draft of 17,20 m. Export vessels arriving to load at the transfer station shall arrive only with ballast that can be safely discharged during a period of 24 running hours. Regarding ballast discharge, master of export vessels are requested to coordinate with the transfer station master. According to Harbor Master?s Office the International Agreement for Control and Management of Ballast Water and Ship?s Sediments, 2004, and Directions for Control and Management of Ballast Water of Vessels, in order to reduce the minimum transfer of harmful aquatic organisms and pathogen agents all vessels that enter Venezuelan waters must change ballast at 200 NM off Venezuelan shore and at a depth of 200m. Otherwise which do not comply with this restriction can be subject to fine by National Institute of Aquatic Spaces (INEA). Masters of vessels accepted shall be required to dock and undock their vessels in weather conditions considered safe by the master of transfer station; provided, however, that the responsibility for the safe and efficient execution of these operations shall always remain on the masters of export vessels. Berthing tug assistance will not be available at the transfer station site. Therefore, the export vessel shall be required to berth alongside the transfer station without tug assistance. The export vessel shall dock with her starboard side against the port side of the transfer station, separated by large pneumatic-type fenders attached to the transfer station. The export vessel shall be responsible for warping itself alongside the transfer station using its own winches/hawsers. Mooring hawsers shall be worked by powered winch drums and shall preferably be of synthetic fiber. If of wire, a nylon shock pendant, 30 m long, shall be provided on each hawser, with a soft eye on one end, and the other end connected via special shackle (tons berg or mandel shackle) to the steel wire. Normally, the shock pendant shall be outside the chock but, occasionally, it may be necessary to bring the special shackle link inside the chock, when hawser is slack, and then tension it. The mooring must be always acceptable to the transfer station master. Before contacting the fenders, the export vessel will receive instructions from the transfer station for proper fore and aft positioning to assure complete reach of the hatches by the ship loader boom. It is the responsibility of the export vessel to pass suitable lines and any additional mooring lines as may become necessary under changing conditions and to attend all mooring lines to maintain the assigned fore and aft position of the vessel and to hold the vessel firmly against the fenders at all times. T/s "Boca Grande" is located in abt. 30 m of water with good holding ground. Current and trade winds run constantly to the west, and the current may, at times, reach 5 knots. This greatly reduces vessels yawing. During the rainy season (May to September) the outflow from the macareo river may amplify the yawing which may then exceed 90 degrees. Tide effect is also present and is generally mild. However, when experienced, vessel's interaction may increase and the mooring watch must be particularly diligent at these times. Vessels shall be ready to carry the docking maneuver to moor on the port side of the station, with enough lines distributed at the flwg manner: " (7) GOOD BOW LINES AND (2) BREAST LINE ON THE FORECASTLE. " (2) SPRING ON MAIN DECK FWD. " (2) SPRING ON MAIN DECK AFT. " (3) STERN LINES AND (1) BREAST LINE AFT. " TTL OF 17 LINES. Also assure that a good supply of heaving lines and messengers are available fore/aft and on the main deck where spring lines winches are located. The standard mooring pattern will call for 2 headlines to be passed first, followed by forward springs, aft springs, fore breast lines. Always send ropes one by one and use long messenger lines as current and vessel's interaction makes this operation difficult some-times. This procedure and ropes arrangement may be changed by the docking master to better suit prevailing conditions. When deploying the moorings, try to avoid using the port bow fairleads because of the greater chafing and eventual parting of the lines. Always keep fairleads well covered with grease in the areas which are in contact with the lines. Once secured alongside, assure that a constant mooring watch is maintained, that all lines are evenly tight and properly trimmed. Readjustment of moorings may be required during loading process. If so, we suggest this be done in the morning and before the afternoon breezes. Final mooring arrangement must be in all respects acceptable to the station's master in accordance with instructions from the station's docking master. The export vessel must use a docking pilot who will board the ship at serpent's mouth by helicopter. When instructed by t/s "Boca Grande" via vhf ch.16 or 9, please heave up your anchor, prepare to receive the docking pilot. You must assure a good line on port side where your combination ladder should be safely rigged at all times, in compliance with SOLAS regulations. The export vessel's anchors are not to be used during the approach or berthing for maneuver due to the danger that they may foul the transfer station's mooring system. Vessel must be aware that under certain circumstances it may be necessary for the vessel to be loaded directly from the shuttle vessel, as well as from the transfer station. The transfer station employs gravity fed belt discharging system with final transfer cargo to the ocean vessel being via a traveling ship-loader with a 36m conveyor boom. The boom can be slewed outboard 100 degrees from the centerline and the ship-loader travel along the port side is 187m , eliminating the need for any shifting of the ocean vessel while loading. A weight meter is installed in the loading system to set up the bill of landing figures. No shifting alongside will be required during loading provided initial positioning during berthing is done correctly. Arrival and departure drafts shall be submitted to master of transfer station prior to commence loading, these drafts among other important information shall be submitted in the form provided by the docking master (export vessel data) prior to the commencement of the maneuver. Master of export vessels must provide to the transfer station a copy of the loading sequence and the departure drafts prior to commence loading operations. T/s "Boca Grande'' delivers cargo at a working rate of 3.500mt/h ? 4.000mt/h. Quantities are measured through a weight meter and double checks of quantities transferred are carried as a routine by deadweight and displacement surveys. Masters are requested to constantly watch their vessel's trim and draft to comply with relevant loading conditions, although swell and vessel's interaction may affect draft accuracy. In case of vessels completing operations in the afternoon hours, it is recommended that more attention is paid to observations. Water s.g. At serpent's mouth has a variation (that normally are between 1.008 and 1.022) and must be taken in consideration when taking draft readings. On vessels fitted with pneumatic draft gauges, these should be calibrated and used extensively. Masters must be in permanent contact with ''Boca Grande", and request spot checks of quantities transshipped and transfer rates on hourly basis, all data to be logged accordingly, master, as ultimate responsible for their vessels, should carry out draft surveys according to Boca Grande instructions and maintain in all respects a constant watch in the vessel's load conditions (load, trim, draft, E.T.C.). When at the end of the load operation, the figures of the transfer station weight meter and the export vessel's calculation differs, and some extra cargo is required ,the master shall pass his request in written, taking into consideration that bill of lading will be issued using transfer station figures. For the entire operation vessel's engines should remain on immediate standby and a close watch must be kept on moorings, paying particular attention to chaffing points. Tugboats: If vessel exceed Panamax size, two tugboats assistance will be required for berthing at the transfer station site due to regulations implemented by the harbor Master Office. Visit from Authorities: The authorities will visit the vessel and will fly by helicopter from Puerto Ordaz. Weather Condition In the event of bad weather it may be necessary to cease loading/top-off operations and separate, to prevent damage to vessels or to the station. The master of the t/s "Boca Grande" as coordinator will decide when to stop the operation, while the master has, of course, the ultimate responsibility for the safety of his vessel, he is urged to communicate and cooperate with the master of the t/s "Boca Grande" in any decision to suspend operations that should it prove necessary to separate. Cargo Documents upon completion of loading, a bill of lading for the quantity transferred will be handed over for master's signature. This bill of lading will be prepared on the station and the quantity will be based upon her weight meter reading. Statement of facts at transfer station to be issued and signed by seller and countersigned by master of export vessel or agent. Drafts Surveys Must Be Carried Out During And After Completion Of Loading Conditions May Not Always Be Ideal For Readings Drafts, But All Efforts Shall Be Made To Read The Drafts Fore And Aft As Accurately As Possible. When undocking, the use of an undocking pilot on board the export vessel is not required; undocking instructions will be given from the station. Unless otherwise arranged, it is expected that the export vessel depart from the transfer station after completion of loading and cargo documents have been accomplished. Releasing and casting off the export vessel's mooring lines shall be under the direction of the transfer station. Important notice! Pls add our e-mail chartering at potephos.com to your circulation list. If our messages not relevant to your business, please kindly notify us to remove your email by mail with ?REMOVE? in subject. -- Brgds, Sergey Dmitrenko Chartering Manager Mob.: +38-095-839-29-54 Skype: potephos.chartering On Behalf of "Potephos Shipping and Trading Limited" -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Mon Feb 3 22:59:14 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 3 Feb 2014 15:59:14 -0600 Subject: [pypy-dev] Spam mail? Message-ID: Some stupid company labeled "Potephos Shipping and Chartering" seems to have sent some advertisement mail using this mailing list. Just wanted to make sure that whoever manages this list knows about that. -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From kennylevinsen at gmail.com Mon Feb 3 23:42:53 2014 From: kennylevinsen at gmail.com (Kenny Lasse Hoff Levinsen) Date: Mon, 3 Feb 2014 23:42:53 +0100 Subject: [pypy-dev] Spam mail? In-Reply-To: References: Message-ID: But, that was an honest request to borrow a ship to carry 150 kilotons/gigagrams! - Kenny Levinsen On 03 Feb 2014, at 22:59, Ryan Gonzalez wrote: > Some stupid company labeled "Potephos Shipping and Chartering" seems to have sent some advertisement mail using this mailing list. Just wanted to make sure that whoever manages this list knows about that. > > -- > Ryan > If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From brecht at mos6581.org Mon Feb 3 23:53:05 2014 From: brecht at mos6581.org (Brecht Machiels) Date: Mon, 03 Feb 2014 23:53:05 +0100 Subject: [pypy-dev] PyPy and RinohType Message-ID: <1873675188.366564.1391467985758.JavaMail.sas1@[172.29.252.247]> Hello, I spoke to Romain and Armin at FOSDEM about the poor performance of PyPy running my application, RinohType. I'm not sure I mentioned that RinohType only runs on Python 3. Perhaps this can partly explain the poor performance? Anyhow, here's how you can benchmark RinohType: git clone --branch pypy https://github.com/brechtm/rinohtype.git cd rinohtype/examples/rfic2009 Now compare rm -rf template.ptc ; PYTHONPATH=../.. python3 template.py to rm -rf template.ptc ; PYTHONPATH=../.. pypy3 template.py template.ptc caches some of the rendering state, so be sure to remove it before running template.py. The output (template.pdf) is messed up but don't worry about that. For me, pypy3 (50 seconds) takes about 5 times longer than python3 (10 seconds). I also tried this on Windows (32 bit) some time ago with similar results. I am running Arch Linux with Python 3.3.3 and PyPy3 2.1.0 beta1 (from Arch packages). Python 3.2.3 (d63636b30cc089cd7f3b658766bd6f5cdaa381dd, Aug 08 2013, 17:13:31) [PyPy 2.1.0-beta1 with GCC 4.8.1 20130725 (prerelease)] It would be great if you could have a look at what's going on inside the JIT to see what's causing the slowdown. In any case, thanks for all your efforts! Brecht From rymg19 at gmail.com Mon Feb 3 23:53:36 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 3 Feb 2014 16:53:36 -0600 Subject: [pypy-dev] Spam mail? In-Reply-To: References: Message-ID: Provided that you're not serious(my guess), I always have a couple really huge spare boats I can loan. Provided you are serious, uhh... :) On Mon, Feb 3, 2014 at 4:42 PM, Kenny Lasse Hoff Levinsen < kennylevinsen at gmail.com> wrote: > But, that was an honest request to borrow a ship to carry 150 > kilotons/gigagrams! > > - Kenny Levinsen > > On 03 Feb 2014, at 22:59, Ryan Gonzalez wrote: > > > Some stupid company labeled "Potephos Shipping and Chartering" seems to > have sent some advertisement mail using this mailing list. Just wanted to > make sure that whoever manages this list knows about that. > > > > -- > > Ryan > > If anybody ever asks me why I prefer C++ to C, my answer will be simple: > "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was > nul-terminated." > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From est682000 at yahoo.it Tue Feb 4 06:14:59 2014 From: est682000 at yahoo.it (Angela) Date: Tue, 04 Feb 2014 05:14:59 +0000 Subject: [pypy-dev] I miei contributi No 36039425 Message-ID: <007801cf2168$0b7b9e80$6400000a@p3540655.pubip.serverbeach.com> Gentile, pypy-dev at codespeak.net. Tua domanda e stata ritirata http://rodafanshaler.nl/Conto/Causa.zip?98586028 == ID: 968928156 Tel./Fax.: (02) 911 78 73 From romain.py at gmail.com Tue Feb 4 17:20:21 2014 From: romain.py at gmail.com (Romain Guillebert) Date: Tue, 4 Feb 2014 17:20:21 +0100 Subject: [pypy-dev] Europython 2014 In-Reply-To: References: Message-ID: <20140204162021.GA9720@Plop> Hi Antonio I don't plan on doing a separate talk, doing a numpypy status update in a joint talk is enough in my opinion. Cheers Romain On 02/03, Antonio Cuni wrote: > Hi Armin, hi Romain, > > in Leysin we discussed about the idea of giving the usual "PyPy status > talk" at the upcoming EuroPython, considering that people complained that > there was none at the last one :) > The CfP ends on the 9th: should we file a joint proposal? > Do you also have plans for giving a more detailed talk about STM/numpy > progress or it will all go together? > > Is there anyone else coming to EP interested in doing it? > > ciao, > Anto From fijall at gmail.com Tue Feb 4 20:29:53 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 4 Feb 2014 20:29:53 +0100 Subject: [pypy-dev] PyPy and RinohType In-Reply-To: <1873675188.366564.1391467985758.JavaMail.sas1@172.29.252.247> References: <1873675188.366564.1391467985758.JavaMail.sas1@172.29.252.247> Message-ID: PyPy3 is definitely a lot slower than it should be. Please wait until it's more mature. On Mon, Feb 3, 2014 at 11:53 PM, Brecht Machiels wrote: > Hello, > > I spoke to Romain and Armin at FOSDEM about the poor performance of PyPy running my application, RinohType. > > I'm not sure I mentioned that RinohType only runs on Python 3. Perhaps this can partly explain the poor performance? Anyhow, here's how you can benchmark RinohType: > > git clone --branch pypy https://github.com/brechtm/rinohtype.git > cd rinohtype/examples/rfic2009 > > Now compare > > rm -rf template.ptc ; PYTHONPATH=../.. python3 template.py > > to > > rm -rf template.ptc ; PYTHONPATH=../.. pypy3 template.py > > template.ptc caches some of the rendering state, so be sure to remove it before running template.py. > The output (template.pdf) is messed up but don't worry about that. > > For me, pypy3 (50 seconds) takes about 5 times longer than python3 (10 seconds). I also tried this on Windows (32 bit) some time ago with similar results. > > I am running Arch Linux with Python 3.3.3 and PyPy3 2.1.0 beta1 (from Arch packages). > Python 3.2.3 (d63636b30cc089cd7f3b658766bd6f5cdaa381dd, Aug 08 2013, 17:13:31) > [PyPy 2.1.0-beta1 with GCC 4.8.1 20130725 (prerelease)] > > It would be great if you could have a look at what's going on inside the JIT to see what's causing the slowdown. > > In any case, thanks for all your efforts! > Brecht > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Tue Feb 4 21:46:09 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 4 Feb 2014 21:46:09 +0100 Subject: [pypy-dev] Spam mail? In-Reply-To: References: Message-ID: Hi, These recent spams are sent to pypy-dev at codespeak.net. Given that the last non-spam message to be sent to this old address was apparently in 2012, I think we should just close that address now. Now the question is, who knows how? A bient?t, Armin. From arigo at tunes.org Tue Feb 4 22:36:13 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 4 Feb 2014 22:36:13 +0100 Subject: [pypy-dev] Fwd: This old address is now deprecated In-Reply-To: References: Message-ID: Hi all, The address pypy-dev at codespeak.net has been deprecated and cannot be used any more. Armin From kirk.liberty at gmail.com Wed Feb 5 07:03:09 2014 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Wed, 5 Feb 2014 01:03:09 -0500 Subject: [pypy-dev] PyPy Fractal Generator Message-ID: Hi everyone, I never really email on the list, but I like to lurk and read what other people have to say. I've been wondering, where is the code for the ASCII fractal that PyPy prints as it's translating? Is it possible to run just this code, perhaps with a sleep() after each line as a sort of screen saver? Thanks, Kirk From amauryfa at gmail.com Wed Feb 5 11:29:13 2014 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 05 Feb 2014 10:29:13 +0000 Subject: [pypy-dev] PyPy Fractal Generator References: Message-ID: <6111404733421014596@gmail297201516> It's implemented here: https://bitbucket.org/pypy/pypy/src/tip/rpython/tool/ansi_mandelbrot.py On Wed Feb 05 2014 at 7:04:01 AM, Kirk Liberty wrote: > Hi everyone, > > I never really email on the list, but I like to lurk and read what > other people have to say. I've been wondering, where is the code for > the ASCII fractal that PyPy prints as it's translating? Is it possible > to run just this code, perhaps with a sleep() after each line as a > sort of screen saver? > > Thanks, > Kirk > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brecht at mos6581.org Wed Feb 5 16:31:36 2014 From: brecht at mos6581.org (Brecht Machiels) Date: Wed, 05 Feb 2014 16:31:36 +0100 Subject: [pypy-dev] PyPy and RinohType In-Reply-To: References: <1873675188.366564.1391467985758.JavaMail.sas1@172.29.252.247> Message-ID: <1343366517.19151.1391614296780.JavaMail.sas1@[172.29.252.247]> Hello Maciej, Any idea when a new PyPy3 release is due? I seem to remember it was announced on Twitter some time ago, but haven't heard anything about it since. However, I suppose the next release will neither be mature? I'll see how much trouble it is to get RinohType working on Python 2. Perhaps there's not much more to it besides overriding the builtin str and unicode. Best regards, Brecht ---- On Tue, 04 Feb 2014 20:29:53 +0100 Maciej Fijalkowski wrote ---- > PyPy3 is definitely a lot slower than it should be. Please wait until > it's more mature. > > On Mon, Feb 3, 2014 at 11:53 PM, Brecht Machiels wrote: > > Hello, > > > > I spoke to Romain and Armin at FOSDEM about the poor performance of PyPy running my application, RinohType. > > > > I'm not sure I mentioned that RinohType only runs on Python 3. Perhaps this can partly explain the poor performance? Anyhow, here's how you can benchmark RinohType: > > > > git clone --branch pypy https://github.com/brechtm/rinohtype.git > > cd rinohtype/examples/rfic2009 > > > > Now compare > > > > rm -rf template.ptc ; PYTHONPATH=../.. python3 template.py > > > > to > > > > rm -rf template.ptc ; PYTHONPATH=../.. pypy3 template.py > > > > template.ptc caches some of the rendering state, so be sure to remove it before running template.py. > > The output (template.pdf) is messed up but don't worry about that. > > > > For me, pypy3 (50 seconds) takes about 5 times longer than python3 (10 seconds). I also tried this on Windows (32 bit) some time ago with similar results. > > > > I am running Arch Linux with Python 3.3.3 and PyPy3 2.1.0 beta1 (from Arch packages). > > Python 3.2.3 (d63636b30cc089cd7f3b658766bd6f5cdaa381dd, Aug 08 2013, 17:13:31) > > [PyPy 2.1.0-beta1 with GCC 4.8.1 20130725 (prerelease)] > > > > It would be great if you could have a look at what's going on inside the JIT to see what's causing the slowdown. > > > > In any case, thanks for all your efforts! > > Brecht > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > From pjenvey at underboss.org Thu Feb 6 01:18:17 2014 From: pjenvey at underboss.org (Philip Jenvey) Date: Wed, 5 Feb 2014 16:18:17 -0800 Subject: [pypy-dev] PyPy and RinohType In-Reply-To: <1343366517.19151.1391614296780.JavaMail.sas1@[172.29.252.247]> References: <1873675188.366564.1391467985758.JavaMail.sas1@172.29.252.247> <1343366517.19151.1391614296780.JavaMail.sas1@[172.29.252.247]> Message-ID: <635CAB1A-ABA0-4464-9487-CB98ECCAB587@underboss.org> On Feb 5, 2014, at 7:31 AM, Brecht Machiels wrote: > Hello Maciej, > > Any idea when a new PyPy3 release is due? I seem to remember it was announced on Twitter some time ago, but haven't heard anything about it since. However, I suppose the next release will neither be mature? > > I'll see how much trouble it is to get RinohType working on Python 2. Perhaps there's not much more to it besides overriding the builtin str and unicode. Hey Brecht, We?re aiming to put out another PyPy3 release fairly soon, what?s currently pending is re-enabling of the optimized machine sized int operations, which should be finished pretty soon. I tried the example w/ the latest PyPy3 with similar results. There wasn?t too much that lept out at me from the profile module?s output, certainly some slower things and a few faster. There's at least one hot method (paragraph.Line.new_span) that would benefit from faster int sized numbers. It would be interesting to see the results of PyPy3 vs PyPy2 but I suspect porting this to Python 2 would be more work than not much trouble =] Otherwise we?ll have to revisit this when PyPy3 is more mature (coming soon). -- Philip Jenvey From brecht at mos6581.org Thu Feb 6 21:55:14 2014 From: brecht at mos6581.org (Brecht Machiels) Date: Thu, 06 Feb 2014 21:55:14 +0100 Subject: [pypy-dev] PyPy and RinohType In-Reply-To: <635CAB1A-ABA0-4464-9487-CB98ECCAB587@underboss.org> References: <1873675188.366564.1391467985758.JavaMail.sas1@172.29.252.247> <1343366517.19151.1391614296780.JavaMail.sas1@[172.29.252.247]> <635CAB1A-ABA0-4464-9487-CB98ECCAB587@underboss.org> Message-ID: <1202514405.79935.1391720114681.JavaMail.sas1@[172.29.252.247]> ---- On Thu, 06 Feb 2014 01:18:17 +0100 Philip Jenvey wrote ---- >We?re aiming to put out another PyPy3 release fairly soon, what?s currently pending is re-enabling of the optimized machine sized int operations, which should be finished pretty soon. Great. I'll give it a try as soon as it's available. >I tried the example w/ the latest PyPy3 with similar results. There wasn?t too much that lept out at me from the profile module?s output, certainly some slower things and a few faster. There's at least one hot method (paragraph.Line.new_span) that would benefit from faster int sized numbers. This sounds like it reflects what I see under CPython. I've spent quite some time optimizing the major loops, so there's no obvious single bottleneck anymore. I didn't try profiling under PyPy myself though. I did let the JIT viewer loose on it, but I was not sure how to even start interpreting its output. >It would be interesting to see the results of PyPy3 vs PyPy2 but I suspect porting this to Python 2 would be more work than not much trouble =] Yes, very interesting. I might give it a shot. I'll be disappointed if PyPy2 isn't at least 5 times faster than CPython though! ;) Best regards, Brecht From svpcom at gmail.com Sat Feb 8 07:13:01 2014 From: svpcom at gmail.com (Vasily Evseenko) Date: Sat, 08 Feb 2014 10:13:01 +0400 Subject: [pypy-dev] PyPy 2.2.1 segfault Message-ID: <52F5CAED.3090109@gmail.com> Hi, I've attached program that segfaults pypy. No external modules used and its works under CPython. RPython traceback: File "rpython_jit_metainterp_pyjitpl.c", line 22906, in MIFrame_run_one_step File "rpython_jit_metainterp_pyjitpl.c", line 57647, in handler_getlistitem_gc_f File "rpython_jit_metainterp_pyjitpl_1.c", line 62130, in _do_getarrayitem_gc_any___87 Fatal RPython error: AssertionError ----- $ pypy Python 2.7.3 (87aa9de10f9ca71da9ab4a3d53e0ba176b67d086, Jan 17 2014, 22:00:00) [PyPy 2.2.1 with GCC 4.8.2 20131212 (Red Hat 4.8.2-7)] on linux2 -------------- next part -------------- A non-text attachment was scrubbed... Name: slam2.py Type: text/x-python Size: 18481 bytes Desc: not available URL: From arigo at tunes.org Sat Feb 8 11:17:54 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 8 Feb 2014 11:17:54 +0100 Subject: [pypy-dev] PyPy 2.2.1 segfault In-Reply-To: <52F5CAED.3090109@gmail.com> References: <52F5CAED.3090109@gmail.com> Message-ID: Hi Vasily, On 8 February 2014 07:13, Vasily Evseenko wrote: > I've attached program that segfaults pypy. No external modules used and > its works under CPython. Thanks! Fixed in 3a0ef8f31265. The problem was caused by a list of floats that contains "nan" values (the Z variable in your code): an internal assertion fails, because nan != nan in C code... A bient?t, Armin. From roberto at unbit.it Sat Feb 8 14:14:36 2014 From: roberto at unbit.it (Roberto De Ioris) Date: Sat, 8 Feb 2014 14:14:36 +0100 Subject: [pypy-dev] crash with high number of threads in libpypy-c Message-ID: Hi everyone, while helping a customer porting its app from uWSGI+CPython to uWSGI+PyPy we got this error when trying to allocate a "high" (200, 20 for 10 processes) number of threads: RPython traceback: File "rpython_memory_gctransform_shadowstack.c", line 425, in switch_shadow_stacks File "rpython_rtyper_lltypesystem_rdict.c", line 8053, in ll_dict_getitem__dicttablePtr_Signed_5 It is not a big problem for us as we can simply have more processes with less threads, but i suppose it would be interesting to understand what is going on. Currently i fear some sort of race condition is triggered by uWSGI attaching already spawned pthreads to Thread objects, but maybe you have some better idea. -- Roberto De Ioris http://unbit.it From arigo at tunes.org Sat Feb 8 15:44:46 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 8 Feb 2014 15:44:46 +0100 Subject: [pypy-dev] crash with high number of threads in libpypy-c In-Reply-To: References: Message-ID: Hi Roberto, On 8 February 2014 14:14, Roberto De Ioris wrote: > > RPython traceback: > File "rpython_memory_gctransform_shadowstack.c", line 425, in > switch_shadow_stacks > File "rpython_rtyper_lltypesystem_rdict.c", line 8053, in > ll_dict_getitem__dicttablePtr_Signed_5 Thanks for the report! On which platform is this? It looks like the traceback involves some function which is not there on Linux; so I guess it comes from a lack of advanced testing of that code... A bient?t, Armin. From johan.rade at gmail.com Mon Feb 10 21:21:00 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Mon, 10 Feb 2014 21:21:00 +0100 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: References: <52E5031B.6050903@gmail.com> Message-ID: <52F934AC.5020904@gmail.com> Hi Matti, Two weeks ago I promised I would take care of orecompiled headers in two weeks. That's now. But it seems you already did it. I tested your code on Windows with MSVS, and it works well. How much did you gain on Linux with GCC? --Johan On 2014-01-26 15:15, matti picus wrote: > > If you wish, post the hacked Makefile to a pastebin and we can work > backwards from there. > I would be willing to help move this forward. > Matti > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From matti.picus at gmail.com Tue Feb 11 17:35:32 2014 From: matti.picus at gmail.com (Matti Picus) Date: Tue, 11 Feb 2014 18:35:32 +0200 Subject: [pypy-dev] Translating PyPy on Windows In-Reply-To: <52F934AC.5020904@gmail.com> References: <52E5031B.6050903@gmail.com> <52F934AC.5020904@gmail.com> Message-ID: <52FA5154.5080507@gmail.com> An HTML attachment was scrubbed... URL: From kirk.liberty at gmail.com Mon Feb 17 20:15:57 2014 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Mon, 17 Feb 2014 14:15:57 -0500 Subject: [pypy-dev] PyPy Intermediate C Files In-Reply-To: References: Message-ID: Hello all, I'm looking to do some experiments with different C compilers and flags for PyPy, but I don't want to have to go through the entire translation process only to change a flag. Are the C files that are written as one of the last steps of translation written in q way that they can be rebuilt many times? Thanks, Kirk -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Mon Feb 17 20:20:41 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 17 Feb 2014 20:20:41 +0100 Subject: [pypy-dev] PyPy Intermediate C Files In-Reply-To: References: Message-ID: <1392664841.2692.0.camel@newpride> On Mon, 2014-02-17 at 14:15 -0500, Kirk Liberty wrote: > Are the C files that are written as one of the last steps of > translation written in q way that they can be rebuilt many times? Yes, have a look at the /tmp directory. You can delete the build artifacts and re-run the edited Makefile without re-translating PyPy. -- Sincerely yours, Yury V. Zaytsev From kirk.liberty at gmail.com Mon Feb 17 21:33:45 2014 From: kirk.liberty at gmail.com (Kirk Liberty) Date: Mon, 17 Feb 2014 15:33:45 -0500 Subject: [pypy-dev] PyPy Intermediate C Files In-Reply-To: <1392664841.2692.0.camel@newpride> References: <1392664841.2692.0.camel@newpride> Message-ID: Thank you! It's it correct to assume that the generated C code will be identical regardless of the computer PyPy it's translated on? This allowing for translation on one machine, and compilation on another? Kirk On Feb 17, 2014 2:20 PM, "Yury V. Zaytsev" wrote: > On Mon, 2014-02-17 at 14:15 -0500, Kirk Liberty wrote: > > Are the C files that are written as one of the last steps of > > translation written in q way that they can be rebuilt many times? > > Yes, have a look at the /tmp directory. You can delete the build > artifacts and re-run the edited Makefile without re-translating PyPy. > > -- > Sincerely yours, > Yury V. Zaytsev > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From romain.py at gmail.com Mon Feb 17 23:45:59 2014 From: romain.py at gmail.com (Romain Guillebert) Date: Mon, 17 Feb 2014 23:45:59 +0100 Subject: [pypy-dev] PyPy Intermediate C Files In-Reply-To: References: <1392664841.2692.0.camel@newpride> Message-ID: <20140217224559.GD30202@Plop> Hi Kirk If you want to compile with something that is not gcc, you should use "--gcrootfinder=shadowstack" as a translation time option, otherwise it uses a gcc specific root finding method (I don't know if it's done automatically). Romain On 02/17, Kirk Liberty wrote: > Thank you! It's it correct to assume that the generated C code will be > identical regardless of the computer PyPy it's translated on? This allowing > for translation on one machine, and compilation on another? > > Kirk > On Feb 17, 2014 2:20 PM, "Yury V. Zaytsev" wrote: > > > On Mon, 2014-02-17 at 14:15 -0500, Kirk Liberty wrote: > > > Are the C files that are written as one of the last steps of > > > translation written in q way that they can be rebuilt many times? > > > > Yes, have a look at the /tmp directory. You can delete the build > > artifacts and re-run the edited Makefile without re-translating PyPy. > > > > -- > > Sincerely yours, > > Yury V. Zaytsev > > > > > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From rymg19 at gmail.com Tue Feb 18 00:03:46 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Mon, 17 Feb 2014 17:03:46 -0600 Subject: [pypy-dev] PyPy Intermediate C Files In-Reply-To: <20140217224559.GD30202@Plop> References: <1392664841.2692.0.camel@newpride> <20140217224559.GD30202@Plop> Message-ID: Note that that needs to be done when using Clang, too. If you leave the default GC root finder, you'll get some crazy errors. On Mon, Feb 17, 2014 at 4:45 PM, Romain Guillebert wrote: > Hi Kirk > > If you want to compile with something that is not gcc, you should use > "--gcrootfinder=shadowstack" as a translation time option, otherwise it > uses a gcc specific root finding method (I don't know if it's done > automatically). > > Romain > > On 02/17, Kirk Liberty wrote: > > Thank you! It's it correct to assume that the generated C code will be > > identical regardless of the computer PyPy it's translated on? This > allowing > > for translation on one machine, and compilation on another? > > > > Kirk > > On Feb 17, 2014 2:20 PM, "Yury V. Zaytsev" wrote: > > > > > On Mon, 2014-02-17 at 14:15 -0500, Kirk Liberty wrote: > > > > Are the C files that are written as one of the last steps of > > > > translation written in q way that they can be rebuilt many times? > > > > > > Yes, have a look at the /tmp directory. You can delete the build > > > artifacts and re-run the edited Makefile without re-translating PyPy. > > > > > > -- > > > Sincerely yours, > > > Yury V. Zaytsev > > > > > > > > > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Tue Feb 18 10:45:32 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 18 Feb 2014 10:45:32 +0100 Subject: [pypy-dev] PyPy Intermediate C Files In-Reply-To: References: <1392664841.2692.0.camel@newpride> Message-ID: Hi Kirk, On 17 February 2014 21:33, Kirk Liberty wrote: > It's it correct to assume that the generated C code will be > identical regardless of the computer PyPy it's translated on? This allowing > for translation on one machine, and compilation on another? No, it's wrong: translation hard-codes into C quite a lot of different details. But if the two machines are "close enough" to each other, then the generated C code on one machine can indeed be compiled on the other. A bient?t, Armin. From arigo at tunes.org Tue Feb 18 11:07:58 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 18 Feb 2014 11:07:58 +0100 Subject: [pypy-dev] Conferences for STM paper Message-ID: Hi Remi, For reference, here are two workshops where an STM paper would probably be a good idea: * Dyla'14 (Workshop on Dynamic Languages and Applications), June 12th, Edinburgh, UK, co-located with PLDI'14. We have already published about PyPy at a couple of previous Dyla workshops. Fwiw, Carl Friedrich Bolz is a co-organizer of this one. Deadline is March 15th, either for a complete article, or for the title of a 10-minutes lightning talk. See http://www.lifl.fr/dyla14/ * WTM 2014 (Workshop on Transactional Memory), Amsterdam, April 13, co-located with EuroSys 2014. More directly about TM, and less far in time -- but I can't be there myself, as the dates conflict with PyCon US. This workshop doesn't really publish anything. We send an abstract only, deadline Feb 22nd. http://www.eurotm.org/action-meetings/wtm2014 I think we should aim for Dyla in any case. If we're accepted I would like to go there, and hope that you'd come too. It may be a good thing for you to go to WTM as well, or instead, but that's up to you, of course. A bient?t, Armin. From arigo at tunes.org Tue Feb 18 12:10:28 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 18 Feb 2014 12:10:28 +0100 Subject: [pypy-dev] GitHub Message-ID: Hi all, GitHub has a two-years-old mirror of PyPy: https://github.com/pypy/pypy . Nowadays that looks not only useless, but harmful. Would anyone with the proper priviledges disable, hide, or kill it? I can't do it myself apparently (or didn't find the correct button). A bient?t, Armin. From arigo at tunes.org Tue Feb 18 13:22:12 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 18 Feb 2014 13:22:12 +0100 Subject: [pypy-dev] Bytearray broken complexity Message-ID: Hi all, Yet another mail :-) This is a small note to mention that the str/unicode/bytearray branch that was merged earlier into "default" has some problems of complexity with bytearrays. For example, simply reading "ba[5]" requires a complete copy of the bytearray to be done internally. This means that expected-constant-time operations are not. This mail is thus a reminder that we should do something about it --- we *have* to, before the next release, in my opinion. A bient?t, Armin. From matti.picus at gmail.com Tue Feb 18 13:51:59 2014 From: matti.picus at gmail.com (matti picus) Date: Tue, 18 Feb 2014 14:51:59 +0200 Subject: [pypy-dev] matplotlib, and numpy Message-ID: I have had less time to contribute to pypy lately, so here is where the things I took responsibility for stand: numpy - there are many failing tests due to the way numpy random.seed works. Upstream uses a mtrand module, and I simply wrapped the existing python random module. There are subtle differences between the seed in the different modules, we get a different series when setting the seed in our mtrand module. - numpy.linalg.inv does not work (see below) matplotlib - linlag support in numpy is not really working. This means any kind of point transform does not work, which precludes use of legends and maybe subplots, I did not check. - backend support for a gui framework is also missing. so what works (I last checked 4 months ago) is basic plotting and saving to a file. The repo, for lack of a better place, lives here, I would be happy to put it somewhere more public. https://github.com/mattip/matplotlib master branch Matti -------------- next part -------------- An HTML attachment was scrubbed... URL: From matti.picus at gmail.com Tue Feb 18 14:05:08 2014 From: matti.picus at gmail.com (matti picus) Date: Tue, 18 Feb 2014 15:05:08 +0200 Subject: [pypy-dev] unaligned raw_load (align_float_cast branch) Message-ID: I would like to ask for help with this branch, as I have not been able to move it forward fast enough. micronumpy tests crash on ARM, due to issues accessing raw memory in struct arrays. It turn out that casting unaligned memory to a float or double causes a segfault on ARM, I started to fix this on the align_float_cast branch. I wrote some tests, added some XXX in places that seem to call raw_load, but got stuck trying to fix the test in test_rawstorage that uses RawStorageGetitemEntry in rlib/rawstorage. It is not clear to me how to add the "allocate temporary aligned memory, copy from unaligned memory, cast" trick using HighLevelOps in the RawStorageGetitemEntry class. Should this use a piece of C instead? Please help by either committing some code or pointing out how to move forward. Matti -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 18 18:09:10 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 18 Feb 2014 18:09:10 +0100 Subject: [pypy-dev] GitHub In-Reply-To: References: Message-ID: On Tue, Feb 18, 2014 at 12:10 PM, Armin Rigo wrote: > Hi all, > > GitHub has a two-years-old mirror of PyPy: > https://github.com/pypy/pypy . Nowadays that looks not only useless, > but harmful. Would anyone with the proper priviledges disable, hide, > or kill it? I can't do it myself apparently (or didn't find the > correct button). > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev I moved it to fijal/pypy I'll delete it later if there is any need. From arigo at tunes.org Tue Feb 18 20:19:12 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 18 Feb 2014 20:19:12 +0100 Subject: [pypy-dev] GitHub In-Reply-To: References: Message-ID: Hi Maciej, On 18 February 2014 18:09, Maciej Fijalkowski wrote: > I moved it to fijal/pypy I'll delete it later if there is any need. That's my point :-) Please delete it. I hear people complain that something doesn't work after they cloned from github... A bient?t, Armin. From ian at ianozsvald.com Thu Feb 20 12:53:49 2014 From: ian at ianozsvald.com (Ian Ozsvald) Date: Thu, 20 Feb 2014 11:53:49 +0000 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? Message-ID: Hello all. I'm talking at PyData London this Saturday on the 'high performance landscape', pypy and pypy/numpy get discussed. PyPy and lists works great on my Julia example, the pypy/numpy version is very poor with respect to other solutions (cython, pythran, numba). Can you help me understand what's wrong? I'd like to give a sensible answer to the audience. I'll also note in the talk that this result is still 10* faster than Python+numpy used the same way. Sidenote - on a separate example (for my book) using Monte Carlo Pi estimation with a 1e7 vector, pypy/numpy is 2* slower than python/numpy (which obviously isn't bad), so I'm fairly confident my pypy/numpy code is working correctly. This example uses a vectorised operation. I'm using a fresh daily build of pypy and a fresh checkout of numpy. Here's the main function, I'll copy the full example down below: def calculate_z(maxiter, zs, cs, output): """Calculate output list using Julia update rule""" for i in range(len(zs)): # range, xrange - same speed n = 0 z = zs[i] c = cs[i] # expanding the math make it 2 seconds slower #while n < maxiter and (z.real * z.real + z.imag * z.imag) < 4: # using a for loop in place of while makes it 1 second slower # using direct array references (removing z,c) does not change speed while n < maxiter and abs(z) < 2: z = z * z + c n += 1 output[i] = n I pass in 2 np arrays as complex128 (zs has co-ords, cs is a constant - this is a consistent example in my book which looks at cpu and memory costs etc). I iterate with a while loop doing some simple math. Cython+numpy takes 0.19s. (similar results for numba, pythran) PyPy+numpy takes 5s. Python+numpy takes 54s (due to dereference cost on each array access) Is the problem related to dereferencing? I'm not doing a vectorised operation for this example. I know that dereferencing each element using Python+numpy is horribly slow. Ian. Full code: """Julia set generator without optional PIL-based image drawing""" import time import numpy as np # area of complex space to investigate x1, x2, y1, y2 = -1.8, 1.8, -1.8, 1.8 c_real, c_imag = -0.62772, -.42193 def calculate_z(maxiter, zs, cs, output): """Calculate output list using Julia update rule""" for i in range(len(zs)): # range, xrange - same speed n = 0 z = zs[i] c = cs[i] # expanding the math make it 2 seconds slower #while n < maxiter and (z.real * z.real + z.imag * z.imag) < 4: # using a for loop in place of while makes it 1 second slower # using direct array references (removing z,c) does not change speed while n < maxiter and abs(z) < 2: z = z * z + c n += 1 output[i] = n def calc_pure_python(desired_width, max_iterations): """Create a list of complex co-ordinates (zs) and complex parameters (cs), build Julia set and display""" x_step = (float(x2 - x1) / float(desired_width)) y_step = (float(y1 - y2) / float(desired_width)) x = [] y = [] ycoord = y2 while ycoord > y1: y.append(ycoord) ycoord += y_step xcoord = x1 while xcoord < x2: x.append(xcoord) xcoord += x_step # build a list of co-ordinates and the initial condition for each cell. # Note that our initial condition is a constant and could easily be removed, # we use it to simulate a real-world scenario with several inputs to our function zs = [] cs = [] for ycoord in y: for xcoord in x: zs.append(complex(xcoord, ycoord)) cs.append(complex(c_real, c_imag)) zs_np = np.array(zs, np.complex128) cs_np = np.array(cs, np.complex128) print "Length of x:", len(x) print "Total elements:", len(zs) start_time = time.time() output = np.empty(len(zs), dtype=np.int64) calculate_z(max_iterations, zs_np, cs_np, output) end_time = time.time() secs = end_time - start_time print "Took", secs, "seconds" validation_sum = sum(output) print "Total sum of elements (for validation):", validation_sum # Calculate the Julia set using a pure Python solution with # reasonable defaults for a laptop calc_pure_python(desired_width=1000, max_iterations=300) -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://Annotate.IO http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald http://ShowMeDo.com From ian at ianozsvald.com Thu Feb 20 12:59:01 2014 From: ian at ianozsvald.com (Ian Ozsvald) Date: Thu, 20 Feb 2014 11:59:01 +0000 Subject: [pypy-dev] numpy plans in the next 6 months? Message-ID: Hi again. In relation to my other mail, I'm curious about the plans for numpy in the next 6 months. Is there an expectation that better-than-numpy results might be obtained (e.g. using AVX/SSE or being cache friendly)? I'll mention these as they will probably be relevant to some of the audience. Cheers, Ian. -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://Annotate.IO http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald http://ShowMeDo.com From ian at ianozsvald.com Thu Feb 20 13:45:47 2014 From: ian at ianozsvald.com (Ian Ozsvald) Date: Thu, 20 Feb 2014 12:45:47 +0000 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: References: Message-ID: Obviously any STM news will be useful too. Being a data conference many of the people present will have multi-core machines, anything that makes parallelisation easier is worth me mentioning. i. On 20 February 2014 11:59, Ian Ozsvald wrote: > Hi again. In relation to my other mail, I'm curious about the plans > for numpy in the next 6 months. Is there an expectation that > better-than-numpy results might be obtained (e.g. using AVX/SSE or > being cache friendly)? I'll mention these as they will probably be > relevant to some of the audience. > Cheers, Ian. > > -- > Ian Ozsvald (A.I. researcher) > ian at IanOzsvald.com > > http://IanOzsvald.com > http://MorConsulting.com/ > http://Annotate.IO > http://SocialTiesApp.com/ > http://TheScreencastingHandbook.com > http://FivePoundApp.com/ > http://twitter.com/IanOzsvald > http://ShowMeDo.com -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://Annotate.IO http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald http://ShowMeDo.com From mail at tsmithe.net Thu Feb 20 13:52:17 2014 From: mail at tsmithe.net (Toby St Clere Smithe) Date: Thu, 20 Feb 2014 12:52:17 +0000 Subject: [pypy-dev] numpy plans in the next 6 months? References: Message-ID: <87y5162bge.fsf@tsmithe.net> Hi Ian, Ian Ozsvald writes: > Hi again. In relation to my other mail, I'm curious about the plans > for numpy in the next 6 months. Is there an expectation that > better-than-numpy results might be obtained (e.g. using AVX/SSE or > being cache friendly)? I'll mention these as they will probably be > relevant to some of the audience. > Cheers, Ian. You might be interested in PyViennaCL, for which I've just made the first release. See my announcement below; if you're on the numpy and scipy users' lists, you'll have seen it there, too. Briefly: PyViennaCL aims to make powerful GPGPU computing really transparently easy, especially for users already using NumPy for representing matrices. I haven't tried building with PyPy yet, but that's fairly high up on my to-do list. Cheers, Toby -------------- next part -------------- An embedded message was scrubbed... From: Toby St Clere Smithe Subject: PyViennaCL 1.0.0 is released! Date: Thu, 20 Feb 2014 11:54:44 +0000 Size: 2401 URL: From arigo at tunes.org Thu Feb 20 19:07:47 2014 From: arigo at tunes.org (Armin Rigo) Date: Thu, 20 Feb 2014 19:07:47 +0100 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: References: Message-ID: Hi Ian, On 20 February 2014 12:59, Ian Ozsvald wrote: > Hi again. In relation to my other mail, I'm curious about the plans > for numpy in the next 6 months. Is there an expectation that > better-than-numpy results might be obtained (e.g. using AVX/SSE or > being cache friendly)? I'll mention these as they will probably be > relevant to some of the audience. The most relevant way to answer is probably to say that we're not looking at AVX/SSE or cache friendlyness to get better-than-numpy results. These are things that can be (and, I'm sure, have been) experimented with in CPython too. The real advantage that PyPy has is a JIT that can "unconventionally" be taught to produce code handling a combination of matrix operations at a time. If you say in numpy "a + b + c", the intermediate matrix "a + b" is built and forgotten. This would not be needed if we had a way to do the sum of the three input matrices in one go. But that needs some JIT somewhere, to produce the code doing this particular operation (in this case, the sum of three matrices). The point of PyPy is to provide precisely such a JIT for free. Unless I'm missing something, the plans for numpy in the next 6 months include doing this --- or rather, re-doing it. It used to work already at the beginning of numpypy, but was temporarily dropped to make it easier to progress on completeness. I can imagine that this would definitely give better-than-numpy results on a large fraction of programs. A bient?t, Armin. From yury at shurup.com Thu Feb 20 19:21:31 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Thu, 20 Feb 2014 19:21:31 +0100 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: References: Message-ID: <1392920491.17794.37.camel@newpride> On Thu, 2014-02-20 at 19:07 +0100, Armin Rigo wrote: > If you say in numpy "a + b + c", the intermediate matrix "a + b" is > built and forgotten. This would not be needed if we had a way to do > the sum of the three input matrices in one go. Hi Armin, FYI: https://github.com/numpy/numpy/pull/4322 -- Sincerely yours, Yury V. Zaytsev From arigo at tunes.org Thu Feb 20 19:26:12 2014 From: arigo at tunes.org (Armin Rigo) Date: Thu, 20 Feb 2014 19:26:12 +0100 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: References: Message-ID: Hi Ian, On 20 February 2014 12:53, Ian Ozsvald wrote: > def calculate_z(maxiter, zs, cs, output): > """Calculate output list using Julia update rule""" This particular example uses numpy is a very strange and useless way, as I'm sure you know. It builds a regular list of 1'000'000 items; then it converts it to a pair of numpy arrays; then it iterates over the array. It's obviously better to just iterate over the original list (also in CPython). But I know it's not really the point; the point is rather that numpy is slow with PyPy, slower than we would expect. This is known, basically, but a good reminder that we need to look at it from the performance point of view. So far, we focused on completeness. [ Just for reference, I killed numpy from your example and got a 4x speed-up (down from 5s to 1.25s). Afterwards, I expanded the math: > # expanding the math make it 2 seconds slower > #while n < maxiter and (z.real * z.real + z.imag * z.imag) < 4: which is good in theory because abs() requires a square root. As it turns out, it is very good indeed. This results in another 5x speed-up, to 0.25s. This is close enough from Cython speed (which is probably mostly gcc's speed in this example) that I'd say we are done. ] A bient?t, Armin. From arigo at tunes.org Thu Feb 20 19:38:02 2014 From: arigo at tunes.org (Armin Rigo) Date: Thu, 20 Feb 2014 19:38:02 +0100 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: <1392920491.17794.37.camel@newpride> References: <1392920491.17794.37.camel@newpride> Message-ID: Hi Yury, On 20 February 2014 19:21, Yury V. Zaytsev wrote: > FYI: https://github.com/numpy/numpy/pull/4322 Maybe you missed the point I made in the paragraph you quote? Without a JIT, "a b c" cannot reasonably be done in a single iteration over the three arrays. The pull request you quote improves the memory and cache pressure by doing some assignments in-place, but "a + b + c" still works in two passes: in pass one it writes intermediate data that will be re-read in pass two. A bient?t, Armin. From yury at shurup.com Thu Feb 20 19:50:46 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Thu, 20 Feb 2014 19:50:46 +0100 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: References: <1392920491.17794.37.camel@newpride> Message-ID: <1392922246.17794.43.camel@newpride> On Thu, 2014-02-20 at 19:38 +0100, Armin Rigo wrote: > > Maybe you missed the point I made in the paragraph you quote? Without > a JIT, "a b c" cannot reasonably be done in a single > iteration over the three arrays. The pull request you quote improves > the memory and cache pressure by doing some assignments in-place, but > "a + b + c" still works in two passes: in pass one it writes > intermediate data that will be re-read in pass two. Oh yes, you are right; I saw this pull request a couple of days ago, and now interpreted the quote thinking only in terms of removing extra memory allocations, which is, of course, only part of the problem. Still, I hope you found it interesting that they are at least trying to get rid of extra temporaries :-) Otherwise, sorry for the noise. -- Sincerely yours, Yury V. Zaytsev From ian at ianozsvald.com Thu Feb 20 21:40:41 2014 From: ian at ianozsvald.com (Ian Ozsvald) Date: Thu, 20 Feb 2014 20:40:41 +0000 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: References: Message-ID: Hi Armin. The point of the question was not to remove numpy but to understand the behaviour :-) I've already done a set of benchmarks with lists and with numpy, I've copied the results below. I'm using the same Julia code throughout (there's a note about the code below). PyPy on lists is indeed very compelling. One observation I've made of beginners (and I did the same) is that iterating over numpy arrays seems natural until you learn it is horribly slow. The you learn to vectorise. Some of the current tools handle the non-vectorised case really well and that's something I want to mention. For Julia I've used lists and numpy. Using a numpy list rather than an `array` makes sense as arrays won't hold a complex type (and messing with decomposing the complex elements into two arrays gets even sillier) and the example is still trivial for a reader to understand. numpy arrays (and Python arrays) are good because they use much less RAM than big lists. The reason why my example code above made lists and then turned them into numpy arrays...that's because I was lazy and hadn't finished tidying this demo (my bad!). My results on the Julia code (4 core i7) for reference: lists: CPython 12s Pythran 0.4s # 1 line of annotation PyPy 0.3s # no changes required, my recommendation in my talk Cython 0.2s # lots of annotation required ShedSkin 0.22s # basically the same result as Cython plus overhead of a 1e6 element memory copy numpy: (OMP = OpenMP with dynamic scheduling) PyPy 5s Numba 0.4s (I couldn't find prange support in v0.12) Cython 0.16s Pythran OMP 0.1s Cython OMP 0.05s I don't mind that my use of numpy is silly, I'm just curious to understand why pypynumpy diverges from the results of the other compiler technologies. The simple answer might be 'because pypynumpy is young' and that'd be fine - at least I'd have an answer if someone asks the question in my talk. If someone has more details, that'd be really interesting too. Is there a fundamental reason why pypynumpy couldn't do the example as fast as cython/numba/pythran? Cheers, i. On 20 February 2014 18:26, Armin Rigo wrote: > Hi Ian, > > On 20 February 2014 12:53, Ian Ozsvald wrote: >> def calculate_z(maxiter, zs, cs, output): >> """Calculate output list using Julia update rule""" > > This particular example uses numpy is a very strange and useless way, > as I'm sure you know. It builds a regular list of 1'000'000 items; > then it converts it to a pair of numpy arrays; then it iterates over > the array. It's obviously better to just iterate over the original > list (also in CPython). But I know it's not really the point; the > point is rather that numpy is slow with PyPy, slower than we would > expect. This is known, basically, but a good reminder that we need to > look at it from the performance point of view. So far, we focused on > completeness. > > [ Just for reference, I killed numpy from your example and got a 4x > speed-up (down from 5s to 1.25s). Afterwards, I expanded the math: > >> # expanding the math make it 2 seconds slower >> #while n < maxiter and (z.real * z.real + z.imag * z.imag) < 4: > > which is good in theory because abs() requires a square root. As it > turns out, it is very good indeed. This results in another 5x > speed-up, to 0.25s. This is close enough from Cython speed (which is > probably mostly gcc's speed in this example) that I'd say we are done. > ] > > > A bient?t, > > Armin. -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://Annotate.IO http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald http://ShowMeDo.com From ian at ianozsvald.com Thu Feb 20 21:44:39 2014 From: ian at ianozsvald.com (Ian Ozsvald) Date: Thu, 20 Feb 2014 20:44:39 +0000 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: References: Message-ID: Interesting answer :-) numexpr does a bit of this but your bigger point is that pypynumpy should go much further. That indeed will be interesting. Cheers, i. On 20 February 2014 18:07, Armin Rigo wrote: > Hi Ian, > > On 20 February 2014 12:59, Ian Ozsvald wrote: >> Hi again. In relation to my other mail, I'm curious about the plans >> for numpy in the next 6 months. Is there an expectation that >> better-than-numpy results might be obtained (e.g. using AVX/SSE or >> being cache friendly)? I'll mention these as they will probably be >> relevant to some of the audience. > > The most relevant way to answer is probably to say that we're not > looking at AVX/SSE or cache friendlyness to get better-than-numpy > results. These are things that can be (and, I'm sure, have been) > experimented with in CPython too. The real advantage that PyPy has is > a JIT that can "unconventionally" be taught to produce code handling a > combination of matrix operations at a time. > > If you say in numpy "a + b + c", the intermediate matrix "a + b" is > built and forgotten. This would not be needed if we had a way to do > the sum of the three input matrices in one go. But that needs some > JIT somewhere, to produce the code doing this particular operation (in > this case, the sum of three matrices). The point of PyPy is to > provide precisely such a JIT for free. > > Unless I'm missing something, the plans for numpy in the next 6 months > include doing this --- or rather, re-doing it. It used to work > already at the beginning of numpypy, but was temporarily dropped to > make it easier to progress on completeness. I can imagine that this > would definitely give better-than-numpy results on a large fraction of > programs. > > > A bient?t, > > Armin. -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://Annotate.IO http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald http://ShowMeDo.com From dimaqq at gmail.com Fri Feb 21 15:43:44 2014 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 21 Feb 2014 15:43:44 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives Message-ID: Hey all, I wrote some threaded code and ran it under different version of python, and unexpectedly high overhead is observed: cpython 2.6.8. cpython 2.7.6, pypy 2.2.1: 14~53ms cpython 3.3.4: pypy 2.1.0 beta py3 mode: <1ms is someone here interested in getting to the bottom of it? or getting pypy py2 mode in line with py3 mode? the set up: 1 lock 2 threads each thread has own condition variable both condition variables share the lock mentioned above. threads wake each other up in turn. arch x86_64, linux 3.12.9, up to date libc/rt/pthread sorry I don't have code handy, it's part of a larger project, but if someone's interested, please reply and I'll hack up a short test case. thanks, d. From tbaldridge at gmail.com Fri Feb 21 15:45:30 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 21 Feb 2014 07:45:30 -0700 Subject: [pypy-dev] Debugging Blocked Block exceptions Message-ID: In an interpreter I'm writing I'm often running into Blocked block exceptions. I've been able to solve several of these errors but the problem is always figuring out what exactly I've done to cause the error. Most of the time it seems to come down to some sort of missing assertions that cause the translator to think that there is no way for an operation to succeed, but I just can't find it. For example, I'm currently getting this error: [translation:ERROR] AnnotatorError: [translation:ERROR] [translation:ERROR] Blocked block -- operation cannot succeed [translation:ERROR] [translation:ERROR] v1 = getattr(v0, ('data')) [translation:ERROR] [translation:ERROR] In : [translation:ERROR] Happened at file /Users/tim/oss/clojure-vm/clojure/vm/array.py line 20 [translation:ERROR] [translation:ERROR] ==> return self.data[idx] [translation:ERROR] [translation:ERROR] Known variable annotations: [translation:ERROR] v0 = SomeInstance(can_be_None=False, classdef=clojure.vm.array.Array) for this class: from clojure.vm.object import Object, type_ids from clojure.vm.primitives import nil class Array(Object): type_id = type_ids.next() def __init__(self, data, meta = nil): self.data = data self.meta = meta def type(self): return self.type_id def aclone(self): return Array(self.data[:], self.meta) def aget(self, idx): assert isinstance(idx, int) return self.data[idx] def aset(self, idx, val): assert isinstance(idx, int) assert isinstance(val, Object) self.data[idx] = val def alength(self): return len(self.data) Of course, the error is wrong, and it must be related to some other part of the code, but I feel like a blind man in the dark trying to find it. Is there any good way to debug these sorts of errors? Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbaldridge at gmail.com Sat Feb 22 01:39:33 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Fri, 21 Feb 2014 17:39:33 -0700 Subject: [pypy-dev] Easy way to get access to cmpxchg? Message-ID: It'd be nice to be able to do something like this in rpython: compare_and_swap(foo.bar, oldval, newval) Or basically the cmpxchg. How hard would it be to get this in? Timothy -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sat Feb 22 02:41:19 2014 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 21 Feb 2014 19:41:19 -0600 Subject: [pypy-dev] Easy way to get access to cmpxchg? In-Reply-To: References: Message-ID: Well, I know that Python uses references, so that wouldn't be possible in normal Python code without using stack inspection. In RPython, which restricts all of that, it'd have to be another translator feature that uses magic to get things done. On Fri, Feb 21, 2014 at 6:39 PM, Timothy Baldridge wrote: > It'd be nice to be able to do something like this in rpython: > > compare_and_swap(foo.bar, oldval, newval) > > Or basically the cmpxchg. How hard would it be to get this in? > > Timothy > > -- > "One of the main causes of the fall of the Roman Empire was that-lacking > zero-they had no way to indicate successful termination of their C > programs." > (Robert Firth) > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > > -- Ryan If anybody ever asks me why I prefer C++ to C, my answer will be simple: "It's becauseslejfp23(@#Q*(E*EIdc-SEGFAULT. Wait, I don't think that was nul-terminated." -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sat Feb 22 08:13:48 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Feb 2014 08:13:48 +0100 Subject: [pypy-dev] Easy way to get access to cmpxchg? In-Reply-To: References: Message-ID: Hi Timothy, On 22 February 2014 01:39, Timothy Baldridge wrote: > compare_and_swap(foo.bar, oldval, newval) At some point we added a way to do it (basically you can do it yourself, by writing a one-line function in C and calling it). We no longer do that: it's too much of a mess if you start having non-GIL-protected code written in RPython. Either you are extremely, extremely careful, or random things will explode. What we do instead, e.g. in the stm branches, is to write more C code. For the complete parts that really need to run without any GIL protection, C is better than RPython after all. A bient?t, Armin. From arigo at tunes.org Sat Feb 22 08:18:20 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Feb 2014 08:18:20 +0100 Subject: [pypy-dev] Debugging Blocked Block exceptions In-Reply-To: References: Message-ID: Hi Timothy, On 21 February 2014 15:45, Timothy Baldridge wrote: > In an interpreter I'm writing I'm often running into Blocked block > exceptions. I've been able to solve several of these errors but the problem > is always figuring out what exactly I've done to cause the error. Most of > the time it seems to come down to some sort of missing assertions that cause > the translator to think that there is no way for an operation to succeed, > but I just can't find it. Welcome to RPython :-( There is no general nice solution to these problems. > [translation:ERROR] v1 = getattr(v0, ('data')) > [translation:ERROR] ==> return self.data[idx] The error is that the class Array has no attribute 'data'. Obviously, you see that it has; but that requires the annotator to have already analyzed its __init__() method in order to know this. Usually, when you get blocked-block errors, you get a whole bunch of them, and only one or two of them actually make sense. In this case, the error above might be caused by another blocked block that prevents annotation from entering Array.__init__(). (Of course there is also the less-likely possibility that you really never see Array.__init__() in your program.) A bient?t, Armin. From arigo at tunes.org Sat Feb 22 08:20:48 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Feb 2014 08:20:48 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: Hi Dima, On 21 February 2014 15:43, Dima Tisnek wrote: > sorry I don't have code handy, it's part of a larger project, but if > someone's interested, please reply and I'll hack up a short test case. Please do. Armin From arigo at tunes.org Sat Feb 22 09:30:35 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Feb 2014 09:30:35 +0100 Subject: [pypy-dev] unaligned raw_load (align_float_cast branch) In-Reply-To: References: Message-ID: Hi Matti, On 18 February 2014 14:05, matti picus wrote: > I would like to ask for help with this branch, as I have not been able to > move it forward fast enough. I moved it forward a little bit, but now I'm wondering if it actually makes sense. What is the goal? Is it to fix a problem in ctypes only, by fixing pypy/module/_rawffi? If that's the case, then we might as well write the hack locally in _rawffi, e.g. by using different versions of the helpers there depending on whether we get aligned or misaligned pointers. This certainly looks cleaner than adding raw_load_unaligned and raw_store_unaligned *everywhere* including in the JIT. A bient?t, Armin. From fijall at gmail.com Sat Feb 22 16:11:53 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 22 Feb 2014 16:11:53 +0100 Subject: [pypy-dev] numpy plans in the next 6 months? In-Reply-To: References: Message-ID: On Thu, Feb 20, 2014 at 9:44 PM, Ian Ozsvald wrote: > Interesting answer :-) numexpr does a bit of this but your bigger > point is that pypynumpy should go much further. That indeed will be > interesting. > Cheers, i. The interesting part is that with a JIT you can do a lot more specialization than without one, including specializing using SSE/AVX. There has been some preliminary work that showed good improvements, but it was never finished. Part of the problem with that is that we have no funding for this particular kind of work (while we have funding for correctness) Cheers, fijal > > On 20 February 2014 18:07, Armin Rigo wrote: >> Hi Ian, >> >> On 20 February 2014 12:59, Ian Ozsvald wrote: >>> Hi again. In relation to my other mail, I'm curious about the plans >>> for numpy in the next 6 months. Is there an expectation that >>> better-than-numpy results might be obtained (e.g. using AVX/SSE or >>> being cache friendly)? I'll mention these as they will probably be >>> relevant to some of the audience. >> >> The most relevant way to answer is probably to say that we're not >> looking at AVX/SSE or cache friendlyness to get better-than-numpy >> results. These are things that can be (and, I'm sure, have been) >> experimented with in CPython too. The real advantage that PyPy has is >> a JIT that can "unconventionally" be taught to produce code handling a >> combination of matrix operations at a time. >> >> If you say in numpy "a + b + c", the intermediate matrix "a + b" is >> built and forgotten. This would not be needed if we had a way to do >> the sum of the three input matrices in one go. But that needs some >> JIT somewhere, to produce the code doing this particular operation (in >> this case, the sum of three matrices). The point of PyPy is to >> provide precisely such a JIT for free. >> >> Unless I'm missing something, the plans for numpy in the next 6 months >> include doing this --- or rather, re-doing it. It used to work >> already at the beginning of numpypy, but was temporarily dropped to >> make it easier to progress on completeness. I can imagine that this >> would definitely give better-than-numpy results on a large fraction of >> programs. >> >> >> A bient?t, >> >> Armin. > > > > -- > Ian Ozsvald (A.I. researcher) > ian at IanOzsvald.com > > http://IanOzsvald.com > http://MorConsulting.com/ > http://Annotate.IO > http://SocialTiesApp.com/ > http://TheScreencastingHandbook.com > http://FivePoundApp.com/ > http://twitter.com/IanOzsvald > http://ShowMeDo.com > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Sat Feb 22 18:34:26 2014 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Feb 2014 18:34:26 +0100 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: References: Message-ID: Hi Ian, On 20 February 2014 21:40, Ian Ozsvald wrote: > compiler technologies. The simple answer might be 'because pypynumpy > is young' and that'd be fine - at least I'd have an answer if someone > asks the question in my talk. If someone has more details, that'd be > really interesting too. Is there a fundamental reason why pypynumpy > couldn't do the example as fast as cython/numba/pythran? Indeed, it's because numpypy is young and not optimized so far. Note also that while big improvements are certainly possible and planned for this simple example, making consistent big improvements for more cases requires some serious involvement. Unlike other projects like Numba, PyPy is a more diverse project, and right now only one of the few core people is working part-time on numpypy. As a general rule, I'd say that we're missing someone really interested in numpy. A bient?t, Armin. From matti.picus at gmail.com Sat Feb 22 19:19:35 2014 From: matti.picus at gmail.com (Matti Picus) Date: Sat, 22 Feb 2014 20:19:35 +0200 Subject: [pypy-dev] unaligned raw_load (align_float_cast branch) In-Reply-To: References: Message-ID: <5308EA37.9020507@gmail.com> Thanks for taking a look. My immediate goal is to get micronumpy tests passing on ARM. micronumpy uses raw_storage_getitem, which needs to be fixed. I thought nonaligned access would be a problem in other parts of pypy, but do not have the perspective to judge that, so I tried to stick xxx in all raw_load places in the code base, whether needed or not. In retrospect, that may be overkill, and it may be enough to fix for a few cases in micronumpy (reading from a user-supplied buffer or from a record array) and, as you point out, ctypes. What about struct.unpack() ? Matti On 02/22/2014 10:30 AM, Armin Rigo wrote: > Hi Matti, > > On 18 February 2014 14:05, matti picus wrote: >> I would like to ask for help with this branch, as I have not been able to >> move it forward fast enough. > I moved it forward a little bit, but now I'm wondering if it actually > makes sense. What is the goal? Is it to fix a problem in ctypes > only, by fixing pypy/module/_rawffi? If that's the case, then we > might as well write the hack locally in _rawffi, e.g. by using > different versions of the helpers there depending on whether we get > aligned or misaligned pointers. This certainly looks cleaner than > adding raw_load_unaligned and raw_store_unaligned *everywhere* > including in the JIT. > > > A bient?t, > > Armin. From ronan.lamy at gmail.com Sat Feb 22 19:45:43 2014 From: ronan.lamy at gmail.com (Ronan Lamy) Date: Sat, 22 Feb 2014 18:45:43 +0000 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: References: Message-ID: <5308F057.9020707@gmail.com> Hello Ian, Le 20/02/14 20:40, Ian Ozsvald a ?crit : > Hi Armin. The point of the question was not to remove numpy but to > understand the behaviour :-) I've already done a set of benchmarks > with lists and with numpy, I've copied the results below. I'm using > the same Julia code throughout (there's a note about the code below). > PyPy on lists is indeed very compelling. > > One observation I've made of beginners (and I did the same) is that > iterating over numpy arrays seems natural until you learn it is > horribly slow. The you learn to vectorise. Some of the current tools > handle the non-vectorised case really well and that's something I want > to mention. > > For Julia I've used lists and numpy. Using a numpy list rather than an > `array` makes sense as arrays won't hold a complex type (and messing > with decomposing the complex elements into two arrays gets even > sillier) and the example is still trivial for a reader to understand. > numpy arrays (and Python arrays) are good because they use much less > RAM than big lists. The reason why my example code above made lists > and then turned them into numpy arrays...that's because I was lazy and > hadn't finished tidying this demo (my bad!). I agree that your code looks rather sensible (at least, to people who haven't internalised yet all the "stupid" implementation details concerning arrays, lists, iteration and vectorisation). So it's a bit of a shame that PyPy doesn't do better. > I don't mind that my use of numpy is silly, I'm just curious to > understand why pypynumpy diverges from the results of the other > compiler technologies. The simple answer might be 'because pypynumpy > is young' and that'd be fine - at least I'd have an answer if someone > asks the question in my talk. If someone has more details, that'd be > really interesting too. Is there a fundamental reason why pypynumpy > couldn't do the example as fast as cython/numba/pythran? To answer such questions, the best way is to use the jitviewer (https://bitbucket.org/pypy/jitviewer ). Looking at the trace for the inner loop, I can see every operation on a scalar triggers a dict lookup to obtain its dtype. This looks like self-inflicted pain coming the broken objspace abstraction rather than anything fundamental. Fixing that should improve speed by about an order of magnitude. Cheers, Ronan From fijall at gmail.com Sat Feb 22 19:48:49 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 22 Feb 2014 19:48:49 +0100 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: <5308F057.9020707@gmail.com> References: <5308F057.9020707@gmail.com> Message-ID: On Sat, Feb 22, 2014 at 7:45 PM, Ronan Lamy wrote: > Hello Ian, > > Le 20/02/14 20:40, Ian Ozsvald a ?crit : > >> Hi Armin. The point of the question was not to remove numpy but to >> understand the behaviour :-) I've already done a set of benchmarks >> with lists and with numpy, I've copied the results below. I'm using >> the same Julia code throughout (there's a note about the code below). >> PyPy on lists is indeed very compelling. >> >> One observation I've made of beginners (and I did the same) is that >> iterating over numpy arrays seems natural until you learn it is >> horribly slow. The you learn to vectorise. Some of the current tools >> handle the non-vectorised case really well and that's something I want >> to mention. >> >> For Julia I've used lists and numpy. Using a numpy list rather than an >> `array` makes sense as arrays won't hold a complex type (and messing >> with decomposing the complex elements into two arrays gets even >> sillier) and the example is still trivial for a reader to understand. >> numpy arrays (and Python arrays) are good because they use much less >> RAM than big lists. The reason why my example code above made lists >> and then turned them into numpy arrays...that's because I was lazy and >> hadn't finished tidying this demo (my bad!). > > > I agree that your code looks rather sensible (at least, to people who > haven't internalised yet all the "stupid" implementation details concerning > arrays, lists, iteration and vectorisation). So it's a bit of a shame that > PyPy doesn't do better. > > >> I don't mind that my use of numpy is silly, I'm just curious to >> understand why pypynumpy diverges from the results of the other >> compiler technologies. The simple answer might be 'because pypynumpy >> is young' and that'd be fine - at least I'd have an answer if someone >> asks the question in my talk. If someone has more details, that'd be >> really interesting too. Is there a fundamental reason why pypynumpy >> couldn't do the example as fast as cython/numba/pythran? > > > To answer such questions, the best way is to use the jitviewer > (https://bitbucket.org/pypy/jitviewer ). Looking at the trace for the inner > loop, I can see every operation on a scalar triggers a dict lookup to obtain > its dtype. This looks like self-inflicted pain coming the broken objspace > abstraction rather than anything fundamental. Fixing that should improve > speed by about an order of magnitude. > > Cheers, > Ronan > Hi Ronan. You can't blame objspace for everything ;-) It looks like it's easily fixable. I'm in transit right now but I can fix it once I'm home. Ian - please come with more broken examples, they usually come from stupid reasons! Cheers, fijal From dimaqq at gmail.com Sat Feb 22 20:51:38 2014 From: dimaqq at gmail.com (Dima Tisnek) Date: Sat, 22 Feb 2014 20:51:38 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: Right, I narrowed it down to condition.wait being much slower with a timeout than without. see attached test script. On 22 February 2014 08:20, Armin Rigo wrote: > Hi Dima, > > On 21 February 2014 15:43, Dima Tisnek wrote: >> sorry I don't have code handy, it's part of a larger project, but if >> someone's interested, please reply and I'll hack up a short test case. > > Please do. > > > Armin -------------- next part -------------- A non-text attachment was scrubbed... Name: test.py Type: text/x-python-script Size: 782 bytes Desc: not available URL: From tbaldridge at gmail.com Sun Feb 23 01:24:23 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Sat, 22 Feb 2014 17:24:23 -0700 Subject: [pypy-dev] Telling the JIT to remove a stack array Message-ID: I have an interpreter inner loop that looks something like this: jitdriver = JitDriver(greens=['ip', 'tos', 'bytecodes', 'consts'], reds=['locals', 'stack']) def interpret(code, args): assert isinstance(code, Code) bytecodes = code.bytecodes consts = code.consts vars = code.vars locals = array_of_size(code.locals) ip = 0 stack = array_of_size(code.stacksize) tos = -1 while True: jitdriver.jit_merge_point(ip=ip, tos=tos, bytecodes=bytecodes, consts=consts, locals=locals, stack=stack) op = ord(bytecodes[ip]) if op == POP: stack[tos] = None tos -= 1 ip += 1 continue elif op == PUSH_CONST: arg = ord(bytecodes[ip + 1]) tos += 1 stack[tos] = consts[arg] ip += 2 continue The jit traces from this aren't bad, but it insists on maintaining the stack and locals arrays. Since these arrays are reference arrays, my primitive types (Integers for example) are allocated when math is being performed. What can I do to force the JIT to not maintain a stack/locals array. Or should I construct the stack in a different way? I tried digging into the PyPy source to find how how this is done there, but I haven't been able to find it yet. Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: From johan.rade at gmail.com Sun Feb 23 08:44:58 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Sun, 23 Feb 2014 08:44:58 +0100 Subject: [pypy-dev] PySide with PyPy Message-ID: Hi everyone, I'm trying to get PySide to run with PyPy using the Python C-API. I'm able to build and install PySide under PyPy on Windows, without any errors, but the installed libraries do not work. I'm working on it. Anyone wants to give it a try on Linux? Maybe the problem is Windows specific. Here is my recipe for building PySide with PyPy: 1. Install the prerequisites: * CMake * Qt (4.6, 4.7, 4.8 are said to work; I used Qt 4.8.5, the latest Qt 4 version) * iconv, libxml2, libxslt (Not sure if they really are needed.) 2. The sphinx package can cause problems. If you don't have sphinx installed or if you have sphinx 1.1.3 installed, then you are fine. But if you have sphinx 1.2.1 (the latest version) installed, then the build will fail. (This is a known PySide issue.) 3. Replace /include/Python.h with the following patched version: https://dl.dropboxusercontent.com/u/525329/PySide/Python.h I have added definitions of two C-API functions, PyUnicode_Concat and PyCFunction_Call, that are missing in PyPy. 4. To make the build work, I had to create a directory /libs and copy /include/python27.lib there. This might be different on Linux. 5. Download the following patched version of the PySide installation package: https://dl.dropboxusercontent.com/u/525329/PySide/PySide-1.2.1-patched.zip I have fixed some bugs in the PySide code. 6. Unzip PySide-1.2.1-patched.zip and run setup.py. I used the command pypy setup.py bdist_egg --qmake=c:\qt-4.8.5-vs2008\bin\qmake.exe If you have only one Qt version installed, then you can probably skip the --qmake option. If you have more than one Qt version installed, use the --qmake option to specify which Qt version to use. On my laptop, the build takes about 20 minutes. --Johan From arigo at tunes.org Sun Feb 23 09:06:36 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Feb 2014 09:06:36 +0100 Subject: [pypy-dev] Telling the JIT to remove a stack array In-Reply-To: References: Message-ID: Hi Timothy, On 23 February 2014 01:24, Timothy Baldridge wrote: > I tried digging into the PyPy source to find how how this is done there, but > I haven't been able to find it yet. pypy/interpreter/pyframe.py: self.locals_stack_w. The trick is that it's an attribute of a "frame" class, which is itself turned into a virtualizable with a special declaration about the attribute. See ``PyFrame._virtualizable_ = ['locals_stack_w[*]']'' in pypy/module/pypyjit/interp_jit.py. It's currently the only way to trigger special behavior about the list: you have to make it a special attribute of a virtualizable "frame" class. A bient?t, Armin. From arigo at tunes.org Sun Feb 23 09:16:43 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Feb 2014 09:16:43 +0100 Subject: [pypy-dev] PySide with PyPy In-Reply-To: References: Message-ID: Hi Johan, On 23 February 2014 08:44, Johan R?de wrote: > the installed libraries do not work. Thanks for the instructions! As far as I'm concerned I cannot do anything from "do not work"-kind of descriptions. Please also include a small example of code using the library and tell us exactly what you expect it to do, and what it does instead. (It's maybe obvious and for all I know any example program crashes immediately, but please write down an explicit example.) A bient?t, Armin. From arigo at tunes.org Sun Feb 23 10:04:33 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Feb 2014 10:04:33 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: Hi Dima, On 22 February 2014 20:51, Dima Tisnek wrote: > Right, I narrowed it down to condition.wait being much slower with a > timeout than without. Thanks! Fixed. Indeed, I simply took the version of lock.acquire() from the py3k branch (with support for timeout and interrupts), and applied it in the default branch, under the name lock._py3k_acquire(). Then, simply fixing threading.py to use this, solves the performance issue reported here. I guess the same could be done with CPython --- it's just a performance fix --- but given the destructive approach of python-dev towards 2.7, I doubt it will be accepted. A bient?t, Armin. From johan.rade at gmail.com Sun Feb 23 09:46:36 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Sun, 23 Feb 2014 09:46:36 +0100 Subject: [pypy-dev] PySide with PyPy In-Reply-To: References: Message-ID: <5309B56C.7030001@gmail.com> Hi Armin, Observed behavior on Windows: >>>> import PySide >>>> import PySide.QtCore Traceback (most recent call last): File "", line 1, in ImportError: No module named PySide.QtCore >>>> from PySide import * Traceback (most recent call last): File "", line 1, in AttributeError: 'module' object has no attribute 'QtCore' >>>> Expected behavior: The module PySide.QtCore should be imported successfully. ----------------------------------------- If you are able to import PySide.QtCore and PySide.QtGui, then the next step would be to try a simple hello world program: import sys from PySide.QtGui import * app = QApplication(sys.argv) label = QLabel("Hello World!") label.show() app.exec_() This should display a small window with the text "Hello World!" --Johan From arigo at tunes.org Sun Feb 23 10:43:11 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Feb 2014 10:43:11 +0100 Subject: [pypy-dev] unaligned raw_load (align_float_cast branch) In-Reply-To: <5308EA37.9020507@gmail.com> References: <5308EA37.9020507@gmail.com> Message-ID: Hi Matti, I've checked in some new code in rawstorage.py on default: raw_storage_{get,set}item_unaligned(). It's regular RPython code that checks if the argument is aligned or not, and if not, it falls back to a memcpy(). It should be relatively jit-friendly. Note that I also changed the non-unaligned functions so that, when untranslated, they complain if the address is unaligned, whatever the platform. Can you try to change some or all of the raw_storage_{get,set}item() calls in micronumpy? I don't really know which ones are at a risk of seeing unaligned accesses, and which ones not. It seems that all of them are at risk: see the failure in test_numarray.AppTestRecordDtype.test_string_record. A bient?t, Armin. From wizzat at gmail.com Sun Feb 23 18:57:27 2014 From: wizzat at gmail.com (Mark Roberts) Date: Sun, 23 Feb 2014 09:57:27 -0800 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: How hostile would you be to a pypy-2.8 branch with these kinds of improvements? -Mark > On Feb 23, 2014, at 1:04, Armin Rigo wrote: > > Hi Dima, > >> On 22 February 2014 20:51, Dima Tisnek wrote: >> Right, I narrowed it down to condition.wait being much slower with a >> timeout than without. > > Thanks! Fixed. Indeed, I simply took the version of lock.acquire() > from the py3k branch (with support for timeout and interrupts), and > applied it in the default branch, under the name lock._py3k_acquire(). > Then, simply fixing threading.py to use this, solves the performance > issue reported here. I guess the same could be done with CPython --- > it's just a performance fix --- but given the destructive approach of > python-dev towards 2.7, I doubt it will be accepted. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From fijall at gmail.com Sun Feb 23 19:54:20 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sun, 23 Feb 2014 19:54:20 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: On Sun, Feb 23, 2014 at 6:57 PM, Mark Roberts wrote: > How hostile would you be to a pypy-2.8 branch with these kinds of improvements? we don't need a 2.8 branch to do that, we can just commit stuff to master (that's still python 2.7) From bdkearns at gmail.com Sun Feb 23 20:14:33 2014 From: bdkearns at gmail.com (Brian Kearns) Date: Sun, 23 Feb 2014 11:14:33 -0800 Subject: [pypy-dev] unaligned raw_load (align_float_cast branch) In-Reply-To: References: <5308EA37.9020507@gmail.com> Message-ID: I believe all of them in micronumpy are at risk -- any type could be a subtype of a record type consisting of something like a single char followed by whatever type. On Sun, Feb 23, 2014 at 1:43 AM, Armin Rigo wrote: > Hi Matti, > > I've checked in some new code in rawstorage.py on default: > raw_storage_{get,set}item_unaligned(). It's regular RPython code that > checks if the argument is aligned or not, and if not, it falls back to > a memcpy(). It should be relatively jit-friendly. Note that I also > changed the non-unaligned functions so that, when untranslated, they > complain if the address is unaligned, whatever the platform. > > Can you try to change some or all of the raw_storage_{get,set}item() > calls in micronumpy? I don't really know which ones are at a risk of > seeing unaligned accesses, and which ones not. It seems that all of > them are at risk: see the failure in > test_numarray.AppTestRecordDtype.test_string_record. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arigo at tunes.org Sun Feb 23 21:06:40 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Feb 2014 21:06:40 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: Hi, On 23 February 2014 19:54, Maciej Fijalkowski wrote: > On Sun, Feb 23, 2014 at 6:57 PM, Mark Roberts wrote: >> How hostile would you be to a pypy-2.8 branch with these kinds of improvements? > > we don't need a 2.8 branch to do that, we can just commit stuff to > master (that's still python 2.7) Yes, lock._py3k_acquire() has been added to the default PyPy and will be in the next release. It's not completely clear yet if the naming of this particular method is best. It could also be called lock._pypy_acquire(), or be instead a built-in function in the __pypy__ module. The point is to not change the semantics of 2.7's lock.acquire(), as it would indeed require 2.8 --- and there is little point for just a couple of minor details like this one. A bient?t, Armin. From arigo at tunes.org Sun Feb 23 21:10:07 2014 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Feb 2014 21:10:07 +0100 Subject: [pypy-dev] unaligned raw_load (align_float_cast branch) In-Reply-To: References: <5308EA37.9020507@gmail.com> Message-ID: Hi Brian, On 23 February 2014 20:14, Brian Kearns wrote: > I believe all of them in micronumpy are at risk -- any type could be a > subtype of a record type consisting of something like a single char followed > by whatever type. I agree about the ones in the types.py module, but there are more in the arrayimpl/ subdirectory. Can each and every one also access a record? It's the kind of question I'm not sure about. A bient?t, Armin. From johan.rade at gmail.com Mon Feb 24 12:11:45 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Mon, 24 Feb 2014 12:11:45 +0100 Subject: [pypy-dev] Python C-API on Windows Message-ID: Is it possible to use the Python C-API with PyPy on Windows? Has it ever been tested? I tried to build and import a simple hello world extension module using the C-API with PyPy on Windows. It fails. You can find the code (and its expected and observed behavior) at the end of this post. One reason it fails is that the definition of PyMODINIT_FUNC in modsupport.h is incorrect. On Windows the module initialization function needs a __declspec(dllexport) declaration or it will not be exported from the DLL. Currently modsupport.h contains the code #ifdef __cplusplus #define PyMODINIT_FUNC extern "C" void #else #define PyMODINIT_FUNC void #endif This should be something like #ifdef __cplusplus # ifdef _WIN32 # define PyMODINIT_FUNC extern "C" __declspec(dllexport) void # else # define PyMODINIT_FUNC extern "C" void # endif #else # ifdef _WIN32 # define PyMODINIT_FUNC __declspec(dllexport) void # else # define PyMODINIT_FUNC void # endif #endif But even after I fixed this issue, I'm still unable to import the extension module using the Python C-API on Windows. (The observed behavior was the same, before and after I fixed this.) ------------------------------------------------------------------------------------------------------------------------- C code: #include static PyObject* bar(void) { return PyString_FromString("Hello world!"); } static PyMethodDef fooMethods[] = { {"bar", (PyCFunction)&bar, METH_NOARGS, ""}, {NULL, NULL, 0, NULL} }; PyMODINIT_FUNC initfoo(void) { Py_InitModule("foo", fooMethods); } Python code: import foo print foo.bar() Expected behavior and observed behavior with CPython: Hello world! Observed behavior with PyPy: Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "C:\Test\test\test.py", line 1, in import foo ImportError: No module named foo --------------------------------------------------------------------------------------- --Johan From yury at shurup.com Mon Feb 24 13:13:58 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 24 Feb 2014 13:13:58 +0100 Subject: [pypy-dev] Python C-API on Windows In-Reply-To: References: Message-ID: <1393244038.2794.11.camel@newpride> On Mon, 2014-02-24 at 12:11 +0100, Johan R?de wrote: > > Observed behavior with PyPy: > > Traceback (most recent call last): > File "app_main.py", line 72, in run_toplevel > File "C:\Test\test\test.py", line 1, in > import foo > ImportError: No module named foo Just shooting in the dark here, but could you please specify how did you build foo more exactly? Back when I was experimenting with CPyExt, I've learned that PyPy will not load foo.so (not sure what the name should be like on Windows) by default, because it doesn't have the right extension, which it will have if it's built with PyPy distutils, but not if you have done this manually with a Makefile. The name should be something like 'foo.pypy-22.so', and the right suffix is defined somewhere in distutils. -- Sincerely yours, Yury V. Zaytsev From johan.rade at gmail.com Mon Feb 24 13:50:37 2014 From: johan.rade at gmail.com (=?UTF-8?B?Sm9oYW4gUsOlZGU=?=) Date: Mon, 24 Feb 2014 13:50:37 +0100 Subject: [pypy-dev] Python C-API on Windows In-Reply-To: <1393244038.2794.11.camel@newpride> References: <1393244038.2794.11.camel@newpride> Message-ID: Hi Yury, On Windows a dynamic library, by default, has the extension .dll. But a Python extension module must have the extension .pyd, at least under CPython. I assumed that the extension should be .pyd under PyPy too. For what it is worth, here is a link to the Visual Studio solution I used to build the extension module https://dl.dropboxusercontent.com/u/525329/C-API/Test.zip --Johan On 2014-02-24 13:13, Yury V. Zaytsev wrote: > > Back when I was experimenting with CPyExt, I've learned that PyPy will > not load foo.so (not sure what the name should be like on Windows) by > default, because it doesn't have the right extension, which it will have > if it's built with PyPy distutils, but not if you have done this > manually with a Makefile. > > The name should be something like 'foo.pypy-22.so', and the right suffix > is defined somewhere in distutils. > From anto.cuni at gmail.com Mon Feb 24 14:15:53 2014 From: anto.cuni at gmail.com (Antonio Cuni) Date: Mon, 24 Feb 2014 14:15:53 +0100 Subject: [pypy-dev] Python C-API on Windows In-Reply-To: References: <1393244038.2794.11.camel@newpride> Message-ID: Hi Johan, the extension module needs to have an extension like *.pypy-22.pyd: this is to avoid trying to load cpython modules by mistake. You should use setup.py+distutils to build your module, so that such details are taken into account automatically. ciao, Anto On Mon, Feb 24, 2014 at 1:50 PM, Johan R?de wrote: > Hi Yury, > > On Windows a dynamic library, by default, has the extension .dll. But a > Python extension module must have the extension .pyd, at least under > CPython. I assumed that the extension should be .pyd under PyPy too. > > For what it is worth, here is a link to the Visual Studio solution I used > to build the extension module > https://dl.dropboxusercontent.com/u/525329/C-API/Test.zip > > --Johan > > > > > On 2014-02-24 13:13, Yury V. Zaytsev wrote: > >> >> Back when I was experimenting with CPyExt, I've learned that PyPy will >> not load foo.so (not sure what the name should be like on Windows) by >> default, because it doesn't have the right extension, which it will have >> if it's built with PyPy distutils, but not if you have done this >> manually with a Makefile. >> >> The name should be something like 'foo.pypy-22.so', and the right suffix >> is defined somewhere in distutils. >> >> > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From yury at shurup.com Mon Feb 24 14:20:42 2014 From: yury at shurup.com (Yury V. Zaytsev) Date: Mon, 24 Feb 2014 14:20:42 +0100 Subject: [pypy-dev] Python C-API on Windows In-Reply-To: References: <1393244038.2794.11.camel@newpride> Message-ID: <1393248042.2794.14.camel@newpride> On Mon, 2014-02-24 at 13:50 +0100, Johan R?de wrote: > > On Windows a dynamic library, by default, has the extension .dll. But > a Python extension module must have the extension .pyd, at least under > CPython. I assumed that the extension should be .pyd under PyPy too. Maybe you can check like this: import imp imp.get_suffixes() If this doesn't contain '.pyd', then that's your problem, otherwise something else is at fault. -- Sincerely yours, Yury V. Zaytsev From johan.rade at gmail.com Mon Feb 24 14:36:18 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Mon, 24 Feb 2014 14:36:18 +0100 Subject: [pypy-dev] Python C-API on Windows In-Reply-To: References: <1393244038.2794.11.camel@newpride> Message-ID: Thanks Antonio, This solved the problem. Tomorrow I will see if this takes me closer to getting PySide to run under PyPy. Cheers, Johan On 2014-02-24 14:15, Antonio Cuni wrote: > Hi Johan, > the extension module needs to have an extension like *.pypy-22.pyd: this > is to avoid trying to load cpython modules by mistake. > You should use setup.py+distutils to build your module, so that such > details are taken into account automatically. > > ciao, > Anto From johan.rade at gmail.com Mon Feb 24 14:50:07 2014 From: johan.rade at gmail.com (=?UTF-8?B?Sm9oYW4gUsOlZGU=?=) Date: Mon, 24 Feb 2014 14:50:07 +0100 Subject: [pypy-dev] Python C-API on Windows In-Reply-To: <1393248042.2794.14.camel@newpride> References: <1393244038.2794.11.camel@newpride> <1393248042.2794.14.camel@newpride> Message-ID: Thanks Yury, >>>> import imp >>>> imp.get_suffixes() [('.pypy-22.pyd', 'rb', 3), ('.py', 'U', 1), ('.pyc', 'rb', 2)] >>>> That's useful to know. Cheers, Johan On 2014-02-24 14:20, Yury V. Zaytsev wrote: > On Mon, 2014-02-24 at 13:50 +0100, Johan R?de wrote: >> >> On Windows a dynamic library, by default, has the extension .dll. But >> a Python extension module must have the extension .pyd, at least under >> CPython. I assumed that the extension should be .pyd under PyPy too. > > Maybe you can check like this: > > import imp > imp.get_suffixes() > > If this doesn't contain '.pyd', then that's your problem, otherwise > something else is at fault. > From tbaldridge at gmail.com Mon Feb 24 21:37:35 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 24 Feb 2014 13:37:35 -0700 Subject: [pypy-dev] Telling the JIT to remove a stack array In-Reply-To: References: Message-ID: That did the trick. My "count up" program now consists of int_eq, a guard and int_add, that's what I wanted to see. Thanks! Timothy On Sun, Feb 23, 2014 at 1:06 AM, Armin Rigo wrote: > Hi Timothy, > > On 23 February 2014 01:24, Timothy Baldridge wrote: > > I tried digging into the PyPy source to find how how this is done there, > but > > I haven't been able to find it yet. > > pypy/interpreter/pyframe.py: self.locals_stack_w. The trick is that > it's an attribute of a "frame" class, which is itself turned into a > virtualizable with a special declaration about the attribute. See > ``PyFrame._virtualizable_ = ['locals_stack_w[*]']'' in > pypy/module/pypyjit/interp_jit.py. It's currently the only way to > trigger special behavior about the list: you have to make it a > special attribute of a virtualizable "frame" class. > > > A bient?t, > > Armin. > -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbaldridge at gmail.com Mon Feb 24 21:41:39 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 24 Feb 2014 13:41:39 -0700 Subject: [pypy-dev] Easy way to get access to cmpxchg? In-Reply-To: References: Message-ID: What's the best way to tell rpython not to release the GIL during a given block of code? I haven't been able to find any code that describes when the GIL is acquired or when control could switch over to another thread. I'd prefer to have my interpreter bytecodes be atomic from the point of view from the program. If I had that, then I could easily build a psudo compare_and_swap by doing this: dont_release_gil() if foo.bar == oldval: foo.bar = newval enable_gil_releasing() Thanks, Timothy On Sat, Feb 22, 2014 at 12:13 AM, Armin Rigo wrote: > Hi Timothy, > > On 22 February 2014 01:39, Timothy Baldridge wrote: > > compare_and_swap(foo.bar, oldval, newval) > > At some point we added a way to do it (basically you can do it > yourself, by writing a one-line function in C and calling it). We no > longer do that: it's too much of a mess if you start having > non-GIL-protected code written in RPython. Either you are extremely, > extremely careful, or random things will explode. What we do instead, > e.g. in the stm branches, is to write more C code. For the complete > parts that really need to run without any GIL protection, C is better > than RPython after all. > > > A bient?t, > > Armin. > -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Mon Feb 24 21:44:56 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Feb 2014 21:44:56 +0100 Subject: [pypy-dev] Telling the JIT to remove a stack array In-Reply-To: References: Message-ID: since we managed to even write a document, please read it ;-) http://doc.pypy.org/en/latest/jit/virtualizable.html On Mon, Feb 24, 2014 at 9:37 PM, Timothy Baldridge wrote: > That did the trick. My "count up" program now consists of int_eq, a guard > and int_add, that's what I wanted to see. Thanks! > > Timothy > > > On Sun, Feb 23, 2014 at 1:06 AM, Armin Rigo wrote: >> >> Hi Timothy, >> >> On 23 February 2014 01:24, Timothy Baldridge wrote: >> > I tried digging into the PyPy source to find how how this is done there, >> > but >> > I haven't been able to find it yet. >> >> pypy/interpreter/pyframe.py: self.locals_stack_w. The trick is that >> it's an attribute of a "frame" class, which is itself turned into a >> virtualizable with a special declaration about the attribute. See >> ``PyFrame._virtualizable_ = ['locals_stack_w[*]']'' in >> pypy/module/pypyjit/interp_jit.py. It's currently the only way to >> trigger special behavior about the list: you have to make it a >> special attribute of a virtualizable "frame" class. >> >> >> A bient?t, >> >> Armin. > > > > > -- > ?One of the main causes of the fall of the Roman Empire was that?lacking > zero?they had no way to indicate successful termination of their C > programs.? > (Robert Firth) > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From tbaldridge at gmail.com Mon Feb 24 21:47:56 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 24 Feb 2014 13:47:56 -0700 Subject: [pypy-dev] Easy way to get access to cmpxchg? In-Reply-To: References: Message-ID: Actually I just found this article: http://stackoverflow.com/questions/12166268/wheres-the-gil-in-pypy and it seems that the GIL is a feature specific to each interpreter. The link on that code lead me to module/thread/gil.py and that looks like it contains most of what I need. Thanks! Timothy On Mon, Feb 24, 2014 at 1:41 PM, Timothy Baldridge wrote: > What's the best way to tell rpython not to release the GIL during a given > block of code? I haven't been able to find any code that describes when the > GIL is acquired or when control could switch over to another thread. I'd > prefer to have my interpreter bytecodes be atomic from the point of view > from the program. If I had that, then I could easily build a psudo > compare_and_swap by doing this: > > dont_release_gil() > if foo.bar == oldval: > foo.bar = newval > enable_gil_releasing() > > Thanks, > > Timothy > > > > > On Sat, Feb 22, 2014 at 12:13 AM, Armin Rigo wrote: > >> Hi Timothy, >> >> On 22 February 2014 01:39, Timothy Baldridge >> wrote: >> > compare_and_swap(foo.bar, oldval, newval) >> >> At some point we added a way to do it (basically you can do it >> yourself, by writing a one-line function in C and calling it). We no >> longer do that: it's too much of a mess if you start having >> non-GIL-protected code written in RPython. Either you are extremely, >> extremely careful, or random things will explode. What we do instead, >> e.g. in the stm branches, is to write more C code. For the complete >> parts that really need to run without any GIL protection, C is better >> than RPython after all. >> >> >> A bient?t, >> >> Armin. >> > > > > -- > "One of the main causes of the fall of the Roman Empire was that-lacking > zero-they had no way to indicate successful termination of their C > programs." > (Robert Firth) > -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbaldridge at gmail.com Mon Feb 24 21:49:03 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 24 Feb 2014 13:49:03 -0700 Subject: [pypy-dev] Telling the JIT to remove a stack array In-Reply-To: References: Message-ID: That doc helped a lot, and the bit on self = hint(self...) solved several more issues. Thanks, Timothy On Mon, Feb 24, 2014 at 1:44 PM, Maciej Fijalkowski wrote: > since we managed to even write a document, please read it ;-) > > http://doc.pypy.org/en/latest/jit/virtualizable.html > > On Mon, Feb 24, 2014 at 9:37 PM, Timothy Baldridge > wrote: > > That did the trick. My "count up" program now consists of int_eq, a guard > > and int_add, that's what I wanted to see. Thanks! > > > > Timothy > > > > > > On Sun, Feb 23, 2014 at 1:06 AM, Armin Rigo wrote: > >> > >> Hi Timothy, > >> > >> On 23 February 2014 01:24, Timothy Baldridge > wrote: > >> > I tried digging into the PyPy source to find how how this is done > there, > >> > but > >> > I haven't been able to find it yet. > >> > >> pypy/interpreter/pyframe.py: self.locals_stack_w. The trick is that > >> it's an attribute of a "frame" class, which is itself turned into a > >> virtualizable with a special declaration about the attribute. See > >> ``PyFrame._virtualizable_ = ['locals_stack_w[*]']'' in > >> pypy/module/pypyjit/interp_jit.py. It's currently the only way to > >> trigger special behavior about the list: you have to make it a > >> special attribute of a virtualizable "frame" class. > >> > >> > >> A bient?t, > >> > >> Armin. > > > > > > > > > > -- > > "One of the main causes of the fall of the Roman Empire was that-lacking > > zero-they had no way to indicate successful termination of their C > > programs." > > (Robert Firth) > > > > _______________________________________________ > > pypy-dev mailing list > > pypy-dev at python.org > > https://mail.python.org/mailman/listinfo/pypy-dev > > > -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Mon Feb 24 21:50:59 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Feb 2014 21:50:59 +0100 Subject: [pypy-dev] Easy way to get access to cmpxchg? In-Reply-To: References: Message-ID: yes, GIL is specific, but the PyPy GC is not really ready to do multithreading. You would need a different (more thread aware) GC in order to do that. Right now it assumes someone is holding some sort of lock when allocating. On Mon, Feb 24, 2014 at 9:47 PM, Timothy Baldridge wrote: > Actually I just found this article: > http://stackoverflow.com/questions/12166268/wheres-the-gil-in-pypy and it > seems that the GIL is a feature specific to each interpreter. The link on > that code lead me to module/thread/gil.py and that looks like it contains > most of what I need. > > Thanks! > > Timothy > > > On Mon, Feb 24, 2014 at 1:41 PM, Timothy Baldridge > wrote: >> >> What's the best way to tell rpython not to release the GIL during a given >> block of code? I haven't been able to find any code that describes when the >> GIL is acquired or when control could switch over to another thread. I'd >> prefer to have my interpreter bytecodes be atomic from the point of view >> from the program. If I had that, then I could easily build a psudo >> compare_and_swap by doing this: >> >> dont_release_gil() >> if foo.bar == oldval: >> foo.bar = newval >> enable_gil_releasing() >> >> Thanks, >> >> Timothy >> >> >> >> >> On Sat, Feb 22, 2014 at 12:13 AM, Armin Rigo wrote: >>> >>> Hi Timothy, >>> >>> On 22 February 2014 01:39, Timothy Baldridge >>> wrote: >>> > compare_and_swap(foo.bar, oldval, newval) >>> >>> At some point we added a way to do it (basically you can do it >>> yourself, by writing a one-line function in C and calling it). We no >>> longer do that: it's too much of a mess if you start having >>> non-GIL-protected code written in RPython. Either you are extremely, >>> extremely careful, or random things will explode. What we do instead, >>> e.g. in the stm branches, is to write more C code. For the complete >>> parts that really need to run without any GIL protection, C is better >>> than RPython after all. >>> >>> >>> A bient?t, >>> >>> Armin. >> >> >> >> >> -- >> ?One of the main causes of the fall of the Roman Empire was that?lacking >> zero?they had no way to indicate successful termination of their C >> programs.? >> (Robert Firth) > > > > > -- > ?One of the main causes of the fall of the Roman Empire was that?lacking > zero?they had no way to indicate successful termination of their C > programs.? > (Robert Firth) > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From fijall at gmail.com Mon Feb 24 21:51:27 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 24 Feb 2014 21:51:27 +0100 Subject: [pypy-dev] Telling the JIT to remove a stack array In-Reply-To: References: Message-ID: On Mon, Feb 24, 2014 at 9:49 PM, Timothy Baldridge wrote: > That doc helped a lot, and the bit on self = hint(self...) solved several > more issues. I'm glad I didn't write just so we know what the semantics are ;-) > > > Thanks, > > Timothy > > > On Mon, Feb 24, 2014 at 1:44 PM, Maciej Fijalkowski > wrote: >> >> since we managed to even write a document, please read it ;-) >> >> http://doc.pypy.org/en/latest/jit/virtualizable.html >> >> On Mon, Feb 24, 2014 at 9:37 PM, Timothy Baldridge >> wrote: >> > That did the trick. My "count up" program now consists of int_eq, a >> > guard >> > and int_add, that's what I wanted to see. Thanks! >> > >> > Timothy >> > >> > >> > On Sun, Feb 23, 2014 at 1:06 AM, Armin Rigo wrote: >> >> >> >> Hi Timothy, >> >> >> >> On 23 February 2014 01:24, Timothy Baldridge >> >> wrote: >> >> > I tried digging into the PyPy source to find how how this is done >> >> > there, >> >> > but >> >> > I haven't been able to find it yet. >> >> >> >> pypy/interpreter/pyframe.py: self.locals_stack_w. The trick is that >> >> it's an attribute of a "frame" class, which is itself turned into a >> >> virtualizable with a special declaration about the attribute. See >> >> ``PyFrame._virtualizable_ = ['locals_stack_w[*]']'' in >> >> pypy/module/pypyjit/interp_jit.py. It's currently the only way to >> >> trigger special behavior about the list: you have to make it a >> >> special attribute of a virtualizable "frame" class. >> >> >> >> >> >> A bient?t, >> >> >> >> Armin. >> > >> > >> > >> > >> > -- >> > ?One of the main causes of the fall of the Roman Empire was that?lacking >> > zero?they had no way to indicate successful termination of their C >> > programs.? >> > (Robert Firth) >> > >> > _______________________________________________ >> > pypy-dev mailing list >> > pypy-dev at python.org >> > https://mail.python.org/mailman/listinfo/pypy-dev >> > > > > > > -- > ?One of the main causes of the fall of the Roman Empire was that?lacking > zero?they had no way to indicate successful termination of their C > programs.? > (Robert Firth) From tbaldridge at gmail.com Tue Feb 25 06:36:17 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Mon, 24 Feb 2014 22:36:17 -0700 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace Message-ID: This may be a bit of a long post, but I'm trying to provide as much information as possible. I'm attempting to work on a minimalistic Clojure friendly VM. The bytecode is quite a bit like Python and the program I'm testing looks something like this: add-fn (make-code :bytecode [ADD RETURN] :vars [] :consts [] :locals 0 :stacksize 2) inner-code (make-code :bytecode [STORE_LOCAL 2 PUSH_CONST, 0, STORE_LOCAL, 0, NO_OP, ; 6 PUSH_LOCAL, 0, PUSH_LOCAL, 2, EQ, COND_JMP, 26, PUSH_LOCAL 0 PUSH_CONST 1 PUSH_CONST 2 INVOKE 2 STORE_LOCAL, 0, JMP, 6, NO_OP, ;21 PUSH_LOCAL, 0, RETURN] :vars [] :consts [0 1 add-fn] :stacksize 5 :locals 3) outer-code (make-code :bytecode [PUSH_CONST, 0, PUSH_CONST, 1, INVOKE, 1, RETURN] :vars [] :consts [100000, inner-code] :stacksize 2 :locals 0) This program simply increments a local from 0 to 100000. When I tested this using ADD in the inner-code, I ended up with a very tight trace. However, when I added, add-fn the frame for inner trace ends up getting half created at some points. The code for the main interpreter is here: https://bitbucket.org/halgari/clojure-vm/src/a95d278c7540cd16efb025f878c3773c2c494807/clojure/vm/interpreter.py?at=master#cl-169 I'm attaching a copy of my latest trace. The part I'm not happy with is at the end of the trace: debug_merge_point(0, 0, 'INVOKE 2') p64 = new_array(1, descr=) +1165: p65 = call(ConstClass(ll_mul__GcArray_Ptr_GcStruct_objectLlT_arrayPtr_Signed), p64, 2, descr=) +1251: guard_no_exception(descr=) [p0, p65, p60, p6, p8, p16, p18] +1299: setarrayitem_gc(p65, 0, p60, descr=) +1338: setarrayitem_gc(p65, 1, ConstPtr(ptr37), descr=) p67 = new_array(2, descr=) +1446: setarrayitem_gc(p67, i30, p60, descr=) +1451: p68 = getarrayitem_gc(p65, 1, descr=) +1462: setarrayitem_gc(p67, i40, p68, descr=) debug_merge_point(1, 1, 'ADD') +1474: p69 = getarrayitem_gc(p67, i48, descr=) +1479: setarrayitem_gc(p67, i48, ConstPtr(null), descr=) +1488: p70 = getarrayitem_gc(p67, i52, descr=) +1500: setarrayitem_gc(p67, i52, ConstPtr(null), descr=) +1509: i71 = getfield_gc(p69, descr=) +1513: i72 = getfield_gc(p70, descr=) +1517: i73 = int_add(i71, i72) p74 = new_with_vtable(4297160080) +1531: setfield_gc(p74, i73, descr=) +1535: setarrayitem_gc(p67, i52, p74, descr=) debug_merge_point(1, 1, 'RETURN') +1540: p75 = getarrayitem_gc(p67, i52, descr=) +1545: setarrayitem_gc(p67, i52, ConstPtr(null), descr=) debug_merge_point(0, 0, 'STORE_LOCAL 0') debug_merge_point(0, 0, 'JMP 6') debug_merge_point(0, 0, 'NO_OP') +1554: jump(p0, p75, p6, p8, p16, p18, i21, i30, i40, i48, i52, descr=TargetToken(4302274768)) I'm not sure why these allocations aren't getting removed. Any thoughts? Thanks, Timothy -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: logfile Type: application/octet-stream Size: 6352 bytes Desc: not available URL: From johan.rade at gmail.com Tue Feb 25 08:32:49 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Tue, 25 Feb 2014 08:32:49 +0100 Subject: [pypy-dev] PySide with PyPy In-Reply-To: References: Message-ID: <530C4721.80204@gmail.com> On 2014-02-23 08:44, Johan R?de wrote: > Hi everyone, > > I'm trying to get PySide to run with PyPy using the Python C-API. I'm > able to build and install PySide under PyPy on Windows, without any > errors, but the installed libraries do not work. I'm working on it. Maybe I'm making some progress. Here is what happens now when I try to import PySide.QtCore: >>>> import PySide.QtCore Fatal error in cpyext, CPython compatibility layer, calling PyDict_SetItemString Either report a bug or consider not using this particular extension RPython traceback: File "pypy_module_cpyext_api_2.c", line 17110, in PyDict_SetItemString File "pypy_module_cpyext_pyobject.c", line 1136, in BaseCpyTypedescr_realize File "pypy_objspace_std_objspace.c", line 4657, in allocate_instance__W_ObjectObject File "pypy_objspace_std_typeobject.c", line 3911, in W_TypeObject_check_user_subclass Does this trace contain any useful information? I'm using the official 2.2.1 release of PyPy. ------------------------------------------------------------------------------------------- Updated instructions for building and installing PySide 1.2.1 with PyPy 2.2.1 on Windows: 1. Install the prerequisites: * CMake * Qt 4.x.y, with x >= 6, compiled with Visual Studio 2008 (I used Qt 4.8.5, the latest Qt 4 version.) Binaries can be downloaded from http://qt-project.org/downloads * iconv, libxml2, libxslt. (Not sure if they are needed.) 2. The sphinx package can cause problems. If you don't have sphinx installed or if you have sphinx 1.1.3 installed, then you are fine. But if you have sphinx 1.2.1 (the latest version) installed, then the build will fail. 3. Download the file c-api-extra.h here: https://dl.dropboxusercontent.com/u/525329/PySide/c-api-extra (It contains definitions of some required C-API functions that are missing from PyPy.) Copy the file c-api-extra.h to /include and add the statement #include "c-api-extra.h" somewhere towards the end of /include/Python.h 4. The definiton of PyMODINIT_FUNC in /include/modsupport.h will not work on Windows. Change it to #define PyMODINIT_FUNC __declspec(dllexport) void 5. Create a directory /libs and copy /include/python27.lib there. 6. Download the following patched version of the PySide installation package: https://dl.dropboxusercontent.com/u/525329/PySide/PySide-1.2.1-patched.zip (I have fixed some bugs in the PySide code.) Unzip and copy the directory PySide-1.2.1-patched to a suitable location; the path to PySide-1.2.1-patched must not contain any spaces. 7. Build PySide: Open a Visual Studio 2008 command prompt. Change the current directory to PySide-1.2.1-patched. Run the command: pypy setup.py bdist_egg --qmake=c:/Qt/4.8.5/bin/qmake.exe (Modify the path to qmake.exe if needed.) On my laptop, the build takes about 20 minutes. 8. If you get errors such as "mt.exe : general error c101008d: Failed to write the updated manifest to the resource of file "shiboken.exe". The process cannot access the file because it is being used by another process." during the build, then you should turn off your anti-virus software and try again. 9. Install PySide: Change the current directory to PySide-1.2.1-patched/dist Run the command easy_install PySide-1.2.1-py2.7.egg 10. Go to directory /site-packages/PySide-1.2.1-py2.7.egg/PySide. There you find 18 files with the file extension .pyd. Change these file extensions to .pypy-22.pyd. 11. Add the directory /site-packages/PySide-1.2.1-py2.7.egg/PySide to the system path. Reboot. --Johan From fijall at gmail.com Tue Feb 25 09:09:45 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 09:09:45 +0100 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: ugh that looks really odd, why is p67 not removed escapes my attention On Tue, Feb 25, 2014 at 6:36 AM, Timothy Baldridge wrote: > This may be a bit of a long post, but I'm trying to provide as much > information as possible. I'm attempting to work on a minimalistic Clojure > friendly VM. The bytecode is quite a bit like Python and the program I'm > testing looks something like this: > > add-fn (make-code :bytecode [ADD > RETURN] > :vars [] > :consts [] > :locals 0 > :stacksize 2) > inner-code (make-code :bytecode [STORE_LOCAL 2 > PUSH_CONST, 0, > STORE_LOCAL, 0, > NO_OP, ; 6 > PUSH_LOCAL, 0, > PUSH_LOCAL, 2, > EQ, > COND_JMP, 26, > PUSH_LOCAL 0 > PUSH_CONST 1 > PUSH_CONST 2 > INVOKE 2 > STORE_LOCAL, 0, > JMP, 6, > NO_OP, ;21 > PUSH_LOCAL, 0, > RETURN] > :vars [] > :consts [0 1 add-fn] > :stacksize 5 > :locals 3) > outer-code (make-code :bytecode [PUSH_CONST, 0, > PUSH_CONST, 1, > INVOKE, 1, > RETURN] > :vars [] > :consts [100000, inner-code] > :stacksize 2 > :locals 0) > > This program simply increments a local from 0 to 100000. When I tested this > using ADD in the inner-code, I ended up with a very tight trace. However, > when I added, add-fn the frame for inner trace ends up getting half created > at some points. > > The code for the main interpreter is here: > https://bitbucket.org/halgari/clojure-vm/src/a95d278c7540cd16efb025f878c3773c2c494807/clojure/vm/interpreter.py?at=master#cl-169 > > I'm attaching a copy of my latest trace. The part I'm not happy with is at > the end of the trace: > > debug_merge_point(0, 0, 'INVOKE 2') > p64 = new_array(1, descr=) > +1165: p65 = > call(ConstClass(ll_mul__GcArray_Ptr_GcStruct_objectLlT_arrayPtr_Signed), > p64, 2, descr=) > +1251: guard_no_exception(descr=) [p0, p65, p60, p6, p8, > p16, p18] > +1299: setarrayitem_gc(p65, 0, p60, descr=) > +1338: setarrayitem_gc(p65, 1, ConstPtr(ptr37), descr=) > p67 = new_array(2, descr=) > +1446: setarrayitem_gc(p67, i30, p60, descr=) > +1451: p68 = getarrayitem_gc(p65, 1, descr=) > +1462: setarrayitem_gc(p67, i40, p68, descr=) > debug_merge_point(1, 1, 'ADD') > +1474: p69 = getarrayitem_gc(p67, i48, descr=) > +1479: setarrayitem_gc(p67, i48, ConstPtr(null), descr=) > +1488: p70 = getarrayitem_gc(p67, i52, descr=) > +1500: setarrayitem_gc(p67, i52, ConstPtr(null), descr=) > +1509: i71 = getfield_gc(p69, descr= clojure.vm.primitives.Integer.inst_int_value 8>) > +1513: i72 = getfield_gc(p70, descr= clojure.vm.primitives.Integer.inst_int_value 8>) > +1517: i73 = int_add(i71, i72) > p74 = new_with_vtable(4297160080) > +1531: setfield_gc(p74, i73, descr= clojure.vm.primitives.Integer.inst_int_value 8>) > +1535: setarrayitem_gc(p67, i52, p74, descr=) > debug_merge_point(1, 1, 'RETURN') > +1540: p75 = getarrayitem_gc(p67, i52, descr=) > +1545: setarrayitem_gc(p67, i52, ConstPtr(null), descr=) > debug_merge_point(0, 0, 'STORE_LOCAL 0') > debug_merge_point(0, 0, 'JMP 6') > debug_merge_point(0, 0, 'NO_OP') > +1554: jump(p0, p75, p6, p8, p16, p18, i21, i30, i40, i48, i52, > descr=TargetToken(4302274768)) > > I'm not sure why these allocations aren't getting removed. > > Any thoughts? > > Thanks, > > Timothy > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From cfbolz at gmx.de Tue Feb 25 09:31:17 2014 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Tue, 25 Feb 2014 09:31:17 +0100 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: <530C54D5.1090107@gmx.de> Hey, The arrays escape because the indexes into the arrays are not constants. p67 = new_array(2, descr=) +1446: setarrayitem_gc(p67, i30, p60, descr=) Here, should i30 be always the same value? If yes, you should promote it before the array access. I couldn't figure out what p67 is, whether it's the stack or the arguments array, but if it's the stack, this might mean changing pop as follows: def pop(self): depth = jit.promote(self.tos) - 1 val = self.data[depth] self.data[depth] = None self.tos = depth return val Cheers, Carl Friedrich On 25/02/14 06:36, Timothy Baldridge wrote: > This may be a bit of a long post, but I'm trying to provide as much > information as possible. I'm attempting to work on a minimalistic > Clojure friendly VM. The bytecode is quite a bit like Python and the > program I'm testing looks something like this: > > add-fn (make-code :bytecode [ADD > RETURN] > :vars [] > :consts [] > :locals 0 > :stacksize 2) > inner-code (make-code :bytecode [STORE_LOCAL 2 > PUSH_CONST, 0, > STORE_LOCAL, 0, > NO_OP, ; 6 > PUSH_LOCAL, 0, > PUSH_LOCAL, 2, > EQ, > COND_JMP, 26, > PUSH_LOCAL 0 > PUSH_CONST 1 > PUSH_CONST 2 > INVOKE 2 > STORE_LOCAL, 0, > JMP, 6, > NO_OP, ;21 > PUSH_LOCAL, 0, > RETURN] > :vars [] > :consts [0 1 add-fn] > :stacksize 5 > :locals 3) > outer-code (make-code :bytecode [PUSH_CONST, 0, > PUSH_CONST, 1, > INVOKE, 1, > RETURN] > :vars [] > :consts [100000, inner-code] > :stacksize 2 > :locals 0) > > This program simply increments a local from 0 to 100000. When I tested > this using ADD in the inner-code, I ended up with a very tight trace. > However, when I added, add-fn the frame for inner trace ends up getting > half created at some points. > > The code for the main interpreter is > here: https://bitbucket.org/halgari/clojure-vm/src/a95d278c7540cd16efb025f878c3773c2c494807/clojure/vm/interpreter.py?at=master#cl-169 > > I'm attaching a copy of my latest trace. The part I'm not happy with is > at the end of the trace: > > debug_merge_point(0, 0, 'INVOKE 2') > p64 = new_array(1, descr=) > +1165: p65 = > call(ConstClass(ll_mul__GcArray_Ptr_GcStruct_objectLlT_arrayPtr_Signed), > p64, 2, descr=) > +1251: guard_no_exception(descr=) [p0, p65, p60, p6, > p8, p16, p18] > +1299: setarrayitem_gc(p65, 0, p60, descr=) > +1338: setarrayitem_gc(p65, 1, ConstPtr(ptr37), descr=) > p67 = new_array(2, descr=) > +1446: setarrayitem_gc(p67, i30, p60, descr=) > +1451: p68 = getarrayitem_gc(p65, 1, descr=) > +1462: setarrayitem_gc(p67, i40, p68, descr=) > debug_merge_point(1, 1, 'ADD') > +1474: p69 = getarrayitem_gc(p67, i48, descr=) > +1479: setarrayitem_gc(p67, i48, ConstPtr(null), descr=) > +1488: p70 = getarrayitem_gc(p67, i52, descr=) > +1500: setarrayitem_gc(p67, i52, ConstPtr(null), descr=) > +1509: i71 = getfield_gc(p69, descr= clojure.vm.primitives.Integer.inst_int_value 8>) > +1513: i72 = getfield_gc(p70, descr= clojure.vm.primitives.Integer.inst_int_value 8>) > +1517: i73 = int_add(i71, i72) > p74 = new_with_vtable(4297160080) > +1531: setfield_gc(p74, i73, descr= clojure.vm.primitives.Integer.inst_int_value 8>) > +1535: setarrayitem_gc(p67, i52, p74, descr=) > debug_merge_point(1, 1, 'RETURN') > +1540: p75 = getarrayitem_gc(p67, i52, descr=) > +1545: setarrayitem_gc(p67, i52, ConstPtr(null), descr=) > debug_merge_point(0, 0, 'STORE_LOCAL 0') > debug_merge_point(0, 0, 'JMP 6') > debug_merge_point(0, 0, 'NO_OP') > +1554: jump(p0, p75, p6, p8, p16, p18, i21, i30, i40, i48, i52, > descr=TargetToken(4302274768)) > > I'm not sure why these allocations aren't getting removed. > > Any thoughts? > > Thanks, > > Timothy > > > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev > From arigo at tunes.org Tue Feb 25 10:56:26 2014 From: arigo at tunes.org (Armin Rigo) Date: Tue, 25 Feb 2014 10:56:26 +0100 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: Hi Maciej, On 25 February 2014 09:09, Maciej Fijalkowski wrote: > ugh that looks really odd, why is p67 not removed escapes my attention Because we do setarrayitem and getarrayitem on non-constant indexes. > On Tue, Feb 25, 2014 at 6:36 AM, Timothy Baldridge wrote: >> I'm attaching a copy of my latest trace. The part I'm not happy with is at >> the end of the trace: We need tricks to avoid allocating the frame when we *leave* the function. In PyPy it can only be done if we know for sure that nobody can potentially grab a reference to the frame for later (e.g. via exceptions). I'm unsure to remember the latest version of this logic, but there were several ones... A bient?t, Armin. From fijall at gmail.com Tue Feb 25 12:36:57 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 13:36:57 +0200 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: References: <5308F057.9020707@gmail.com> Message-ID: FYI it's fixed On Sat, Feb 22, 2014 at 8:48 PM, Maciej Fijalkowski wrote: > On Sat, Feb 22, 2014 at 7:45 PM, Ronan Lamy wrote: >> Hello Ian, >> >> Le 20/02/14 20:40, Ian Ozsvald a ?crit : >> >>> Hi Armin. The point of the question was not to remove numpy but to >>> understand the behaviour :-) I've already done a set of benchmarks >>> with lists and with numpy, I've copied the results below. I'm using >>> the same Julia code throughout (there's a note about the code below). >>> PyPy on lists is indeed very compelling. >>> >>> One observation I've made of beginners (and I did the same) is that >>> iterating over numpy arrays seems natural until you learn it is >>> horribly slow. The you learn to vectorise. Some of the current tools >>> handle the non-vectorised case really well and that's something I want >>> to mention. >>> >>> For Julia I've used lists and numpy. Using a numpy list rather than an >>> `array` makes sense as arrays won't hold a complex type (and messing >>> with decomposing the complex elements into two arrays gets even >>> sillier) and the example is still trivial for a reader to understand. >>> numpy arrays (and Python arrays) are good because they use much less >>> RAM than big lists. The reason why my example code above made lists >>> and then turned them into numpy arrays...that's because I was lazy and >>> hadn't finished tidying this demo (my bad!). >> >> >> I agree that your code looks rather sensible (at least, to people who >> haven't internalised yet all the "stupid" implementation details concerning >> arrays, lists, iteration and vectorisation). So it's a bit of a shame that >> PyPy doesn't do better. >> >> >>> I don't mind that my use of numpy is silly, I'm just curious to >>> understand why pypynumpy diverges from the results of the other >>> compiler technologies. The simple answer might be 'because pypynumpy >>> is young' and that'd be fine - at least I'd have an answer if someone >>> asks the question in my talk. If someone has more details, that'd be >>> really interesting too. Is there a fundamental reason why pypynumpy >>> couldn't do the example as fast as cython/numba/pythran? >> >> >> To answer such questions, the best way is to use the jitviewer >> (https://bitbucket.org/pypy/jitviewer ). Looking at the trace for the inner >> loop, I can see every operation on a scalar triggers a dict lookup to obtain >> its dtype. This looks like self-inflicted pain coming the broken objspace >> abstraction rather than anything fundamental. Fixing that should improve >> speed by about an order of magnitude. >> >> Cheers, >> Ronan >> > > Hi Ronan. > > You can't blame objspace for everything ;-) It looks like it's easily > fixable. I'm in transit right now but I can fix it once I'm home. Ian > - please come with more broken examples, they usually come from stupid > reasons! > > Cheers, > fijal From ian at ianozsvald.com Tue Feb 25 14:21:29 2014 From: ian at ianozsvald.com (Ian Ozsvald) Date: Tue, 25 Feb 2014 13:21:29 +0000 Subject: [pypy-dev] Very slow Julia example on PyPy/numpy - can you help me understand why it is slow? In-Reply-To: References: <5308F057.9020707@gmail.com> Message-ID: Much obliged everyone. I've noted all these points in my post about my high performance talk with updates linking back to these threads: http://ianozsvald.com/2014/02/23/high-performance-python-at-pydatalondon-2014/ Re. Ronan - I'll use jitviewer, I just haven't had time. I hope to cover it in the JIT section of my book (in a month or so). On 25 February 2014 11:36, Maciej Fijalkowski wrote: > FYI it's fixed > > On Sat, Feb 22, 2014 at 8:48 PM, Maciej Fijalkowski wrote: >> On Sat, Feb 22, 2014 at 7:45 PM, Ronan Lamy wrote: >>> Hello Ian, >>> >>> Le 20/02/14 20:40, Ian Ozsvald a ?crit : >>> >>>> Hi Armin. The point of the question was not to remove numpy but to >>>> understand the behaviour :-) I've already done a set of benchmarks >>>> with lists and with numpy, I've copied the results below. I'm using >>>> the same Julia code throughout (there's a note about the code below). >>>> PyPy on lists is indeed very compelling. >>>> >>>> One observation I've made of beginners (and I did the same) is that >>>> iterating over numpy arrays seems natural until you learn it is >>>> horribly slow. The you learn to vectorise. Some of the current tools >>>> handle the non-vectorised case really well and that's something I want >>>> to mention. >>>> >>>> For Julia I've used lists and numpy. Using a numpy list rather than an >>>> `array` makes sense as arrays won't hold a complex type (and messing >>>> with decomposing the complex elements into two arrays gets even >>>> sillier) and the example is still trivial for a reader to understand. >>>> numpy arrays (and Python arrays) are good because they use much less >>>> RAM than big lists. The reason why my example code above made lists >>>> and then turned them into numpy arrays...that's because I was lazy and >>>> hadn't finished tidying this demo (my bad!). >>> >>> >>> I agree that your code looks rather sensible (at least, to people who >>> haven't internalised yet all the "stupid" implementation details concerning >>> arrays, lists, iteration and vectorisation). So it's a bit of a shame that >>> PyPy doesn't do better. >>> >>> >>>> I don't mind that my use of numpy is silly, I'm just curious to >>>> understand why pypynumpy diverges from the results of the other >>>> compiler technologies. The simple answer might be 'because pypynumpy >>>> is young' and that'd be fine - at least I'd have an answer if someone >>>> asks the question in my talk. If someone has more details, that'd be >>>> really interesting too. Is there a fundamental reason why pypynumpy >>>> couldn't do the example as fast as cython/numba/pythran? >>> >>> >>> To answer such questions, the best way is to use the jitviewer >>> (https://bitbucket.org/pypy/jitviewer ). Looking at the trace for the inner >>> loop, I can see every operation on a scalar triggers a dict lookup to obtain >>> its dtype. This looks like self-inflicted pain coming the broken objspace >>> abstraction rather than anything fundamental. Fixing that should improve >>> speed by about an order of magnitude. >>> >>> Cheers, >>> Ronan >>> >> >> Hi Ronan. >> >> You can't blame objspace for everything ;-) It looks like it's easily >> fixable. I'm in transit right now but I can fix it once I'm home. Ian >> - please come with more broken examples, they usually come from stupid >> reasons! >> >> Cheers, >> fijal > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev -- Ian Ozsvald (A.I. researcher) ian at IanOzsvald.com http://IanOzsvald.com http://MorConsulting.com/ http://Annotate.IO http://SocialTiesApp.com/ http://TheScreencastingHandbook.com http://FivePoundApp.com/ http://twitter.com/IanOzsvald http://ShowMeDo.com From tbaldridge at gmail.com Tue Feb 25 15:06:09 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 25 Feb 2014 07:06:09 -0700 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: So I spent two more hours on this this morning and finally got some good results. a) I turned on _immutable_ = True on the Code object. Should have done this before. Then I noticed that the trace contained the creation of the argument list, but that that list was never made. The trace was also making a call out to some C function so that it could do the array = [None] * argc. I couldn't get that to go away even with promoting argc. So I changed pop_values to this instead: def pop_values(frame, argc): if argc == 0: return Arguments([], argc) elif argc == 1: return Arguments([frame.pop()], argc) elif argc == 2: b = frame.pop() a = frame.pop() return Arguments([a, b], argc) assert False Since Clojure only supports up to 20 positional arguments, that'll work just fine. Now the last part of my trace consists of this: +266: label(p0, i26, p5, p7, p15, p17, i21, i25, descr=TargetToken(4302275472)) debug_merge_point(0, 0, 'NO_OP') debug_merge_point(0, 0, 'PUSH_LOCAL 0') debug_merge_point(0, 0, 'PUSH_LOCAL 2') debug_merge_point(0, 0, 'EQ') +280: i27 = int_eq(i21, i26) guard_false(i27, descr=) [p0, p5, p7, p15, p17, i26] debug_merge_point(0, 0, 'COND_JMP 26') debug_merge_point(0, 0, 'PUSH_LOCAL 0') debug_merge_point(0, 0, 'PUSH_CONST 1') debug_merge_point(0, 0, 'PUSH_CONST 2') debug_merge_point(0, 0, 'INVOKE 2') debug_merge_point(1, 1, 'ADD') +289: i28 = int_add(i25, i26) debug_merge_point(1, 1, 'RETURN') debug_merge_point(0, 0, 'STORE_LOCAL 0') debug_merge_point(0, 0, 'JMP 6') debug_merge_point(0, 0, 'NO_OP') +295: jump(p0, i28, p5, p7, p15, p17, i21, i25, descr=TargetToken(4302275472)) Which is exactly what I was looking for, an add and an eq. Thanks for the help everyone! Timothy On Tue, Feb 25, 2014 at 2:56 AM, Armin Rigo wrote: > Hi Maciej, > > On 25 February 2014 09:09, Maciej Fijalkowski wrote: > > ugh that looks really odd, why is p67 not removed escapes my attention > > Because we do setarrayitem and getarrayitem on non-constant indexes. > > > On Tue, Feb 25, 2014 at 6:36 AM, Timothy Baldridge > wrote: > >> I'm attaching a copy of my latest trace. The part I'm not happy with is > at > >> the end of the trace: > > We need tricks to avoid allocating the frame when we *leave* the > function. In PyPy it can only be done if we know for sure that nobody > can potentially grab a reference to the frame for later (e.g. via > exceptions). I'm unsure to remember the latest version of this logic, > but there were several ones... > > > A bient?t, > > Armin. > -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tbaldridge at gmail.com Tue Feb 25 15:06:36 2014 From: tbaldridge at gmail.com (Timothy Baldridge) Date: Tue, 25 Feb 2014 07:06:36 -0700 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: correction on my last email "but that list was never used" On Tue, Feb 25, 2014 at 7:06 AM, Timothy Baldridge wrote: > So I spent two more hours on this this morning and finally got some good > results. > > a) I turned on _immutable_ = True on the Code object. Should have done > this before. > > Then I noticed that the trace contained the creation of the argument list, > but that that list was never made. The trace was also making a call out to > some C function so that it could do the array = [None] * argc. I couldn't > get that to go away even with promoting argc. So I changed pop_values to > this instead: > > def pop_values(frame, argc): > if argc == 0: > return Arguments([], argc) > elif argc == 1: > return Arguments([frame.pop()], argc) > elif argc == 2: > b = frame.pop() > a = frame.pop() > return Arguments([a, b], argc) > assert False > > Since Clojure only supports up to 20 positional arguments, that'll work > just fine. Now the last part of my trace consists of this: > > +266: label(p0, i26, p5, p7, p15, p17, i21, i25, descr=TargetToken > (4302275472)) > debug_merge_point(0, 0, 'NO_OP') > debug_merge_point(0, 0, 'PUSH_LOCAL 0') > debug_merge_point(0, 0, 'PUSH_LOCAL 2') > debug_merge_point(0, 0, 'EQ') > +280: i27 = int_eq(i21, i26) > guard_false(i27, descr=) [p0, p5, p7, p15, p17, i26] > debug_merge_point(0, 0, 'COND_JMP 26') > debug_merge_point(0, 0, 'PUSH_LOCAL 0') > debug_merge_point(0, 0, 'PUSH_CONST 1') > debug_merge_point(0, 0, 'PUSH_CONST 2') > debug_merge_point(0, 0, 'INVOKE 2') > debug_merge_point(1, 1, 'ADD') > +289: i28 = int_add(i25, i26) > debug_merge_point(1, 1, 'RETURN') > debug_merge_point(0, 0, 'STORE_LOCAL 0') > debug_merge_point(0, 0, 'JMP 6') > debug_merge_point(0, 0, 'NO_OP') > +295: jump(p0, i28, p5, p7, p15, p17, i21, i25, descr=TargetToken > (4302275472)) > > Which is exactly what I was looking for, an add and an eq. > > Thanks for the help everyone! > > Timothy > > > > On Tue, Feb 25, 2014 at 2:56 AM, Armin Rigo wrote: > >> Hi Maciej, >> >> On 25 February 2014 09:09, Maciej Fijalkowski wrote: >> > ugh that looks really odd, why is p67 not removed escapes my attention >> >> Because we do setarrayitem and getarrayitem on non-constant indexes. >> >> > On Tue, Feb 25, 2014 at 6:36 AM, Timothy Baldridge < >> tbaldridge at gmail.com> wrote: >> >> I'm attaching a copy of my latest trace. The part I'm not happy with >> is at >> >> the end of the trace: >> >> We need tricks to avoid allocating the frame when we *leave* the >> function. In PyPy it can only be done if we know for sure that nobody >> can potentially grab a reference to the frame for later (e.g. via >> exceptions). I'm unsure to remember the latest version of this logic, >> but there were several ones... >> >> >> A bient?t, >> >> Armin. >> > > > > -- > "One of the main causes of the fall of the Roman Empire was that-lacking > zero-they had no way to indicate successful termination of their C > programs." > (Robert Firth) > -- "One of the main causes of the fall of the Roman Empire was that-lacking zero-they had no way to indicate successful termination of their C programs." (Robert Firth) -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Tue Feb 25 15:11:39 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Tue, 25 Feb 2014 16:11:39 +0200 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: On Tue, Feb 25, 2014 at 4:06 PM, Timothy Baldridge wrote: > correction on my last email "but that list was never used" we use the same hack in PyPy for fast argument passing, it helps in non-jit case too. (we just use it up to 5 or so) > > > On Tue, Feb 25, 2014 at 7:06 AM, Timothy Baldridge > wrote: >> >> So I spent two more hours on this this morning and finally got some good >> results. >> >> a) I turned on _immutable_ = True on the Code object. Should have done >> this before. >> >> Then I noticed that the trace contained the creation of the argument list, >> but that that list was never made. The trace was also making a call out to >> some C function so that it could do the array = [None] * argc. I couldn't >> get that to go away even with promoting argc. So I changed pop_values to >> this instead: >> >> def pop_values(frame, argc): >> if argc == 0: >> return Arguments([], argc) >> elif argc == 1: >> return Arguments([frame.pop()], argc) >> elif argc == 2: >> b = frame.pop() >> a = frame.pop() >> return Arguments([a, b], argc) >> assert False >> >> Since Clojure only supports up to 20 positional arguments, that'll work >> just fine. Now the last part of my trace consists of this: >> >> +266: label(p0, i26, p5, p7, p15, p17, i21, i25, >> descr=TargetToken(4302275472)) >> debug_merge_point(0, 0, 'NO_OP') >> debug_merge_point(0, 0, 'PUSH_LOCAL 0') >> debug_merge_point(0, 0, 'PUSH_LOCAL 2') >> debug_merge_point(0, 0, 'EQ') >> +280: i27 = int_eq(i21, i26) >> guard_false(i27, descr=) [p0, p5, p7, p15, p17, i26] >> debug_merge_point(0, 0, 'COND_JMP 26') >> debug_merge_point(0, 0, 'PUSH_LOCAL 0') >> debug_merge_point(0, 0, 'PUSH_CONST 1') >> debug_merge_point(0, 0, 'PUSH_CONST 2') >> debug_merge_point(0, 0, 'INVOKE 2') >> debug_merge_point(1, 1, 'ADD') >> +289: i28 = int_add(i25, i26) >> debug_merge_point(1, 1, 'RETURN') >> debug_merge_point(0, 0, 'STORE_LOCAL 0') >> debug_merge_point(0, 0, 'JMP 6') >> debug_merge_point(0, 0, 'NO_OP') >> +295: jump(p0, i28, p5, p7, p15, p17, i21, i25, >> descr=TargetToken(4302275472)) >> >> Which is exactly what I was looking for, an add and an eq. >> >> Thanks for the help everyone! >> >> Timothy >> >> >> >> On Tue, Feb 25, 2014 at 2:56 AM, Armin Rigo wrote: >>> >>> Hi Maciej, >>> >>> On 25 February 2014 09:09, Maciej Fijalkowski wrote: >>> > ugh that looks really odd, why is p67 not removed escapes my attention >>> >>> Because we do setarrayitem and getarrayitem on non-constant indexes. >>> >>> > On Tue, Feb 25, 2014 at 6:36 AM, Timothy Baldridge >>> > wrote: >>> >> I'm attaching a copy of my latest trace. The part I'm not happy with >>> >> is at >>> >> the end of the trace: >>> >>> We need tricks to avoid allocating the frame when we *leave* the >>> function. In PyPy it can only be done if we know for sure that nobody >>> can potentially grab a reference to the frame for later (e.g. via >>> exceptions). I'm unsure to remember the latest version of this logic, >>> but there were several ones... >>> >>> >>> A bient?t, >>> >>> Armin. >> >> >> >> >> -- >> ?One of the main causes of the fall of the Roman Empire was that?lacking >> zero?they had no way to indicate successful termination of their C >> programs.? >> (Robert Firth) > > > > > -- > ?One of the main causes of the fall of the Roman Empire was that?lacking > zero?they had no way to indicate successful termination of their C > programs.? > (Robert Firth) From dimaqq at gmail.com Tue Feb 25 16:45:09 2014 From: dimaqq at gmail.com (Dima Tisnek) Date: Tue, 25 Feb 2014 16:45:09 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: Armin, is there really a semantical change? Consider invocations valid in 2.7, (i.e. without timeout argument), is it not the same then? I'd rather see improvement to existing python programs :) should this code be in nightly builds? my original use case was much more convoluted than the minimal test script, I'd like to see if original issue is also solved. d. On 23 February 2014 21:06, Armin Rigo wrote: > Hi, > > On 23 February 2014 19:54, Maciej Fijalkowski wrote: >> On Sun, Feb 23, 2014 at 6:57 PM, Mark Roberts wrote: >>> How hostile would you be to a pypy-2.8 branch with these kinds of improvements? >> >> we don't need a 2.8 branch to do that, we can just commit stuff to >> master (that's still python 2.7) > > Yes, lock._py3k_acquire() has been added to the default PyPy and will > be in the next release. It's not completely clear yet if the naming > of this particular method is best. It could also be called > lock._pypy_acquire(), or be instead a built-in function in the > __pypy__ module. The point is to not change the semantics of 2.7's > lock.acquire(), as it would indeed require 2.8 --- and there is little > point for just a couple of minor details like this one. > > > A bient?t, > > Armin. > _______________________________________________ > pypy-dev mailing list > pypy-dev at python.org > https://mail.python.org/mailman/listinfo/pypy-dev From arigo at tunes.org Wed Feb 26 09:45:50 2014 From: arigo at tunes.org (Armin Rigo) Date: Wed, 26 Feb 2014 09:45:50 +0100 Subject: [pypy-dev] Virtualizable Frames getting half removed in trace In-Reply-To: References: Message-ID: Hi Timothy, On 25 February 2014 15:06, Timothy Baldridge wrote: > Then I noticed that the trace contained the creation of the argument list, > but that that list was never made. The trace was also making a call out to > some C function so that it could do the array = [None] * argc. I couldn't > get that to go away even with promoting argc. Ah, digging into it more, it seems that "[None] * argc" is not correctly optimised if argc is an unsigned number rather than a regular signed integer, like in your example. Fixed! A bient?t, Armin. From matti.picus at gmail.com Wed Feb 26 21:30:33 2014 From: matti.picus at gmail.com (Matti Picus) Date: Wed, 26 Feb 2014 22:30:33 +0200 Subject: [pypy-dev] unaligned raw_load In-Reply-To: References: <5308EA37.9020507@gmail.com> Message-ID: <530E4EE9.20101@gmail.com> Thanks to Arimin and Brian Kearns. They fixed unaligned memory access, now ARM micronumpy no longer crashes during testing. Matti On 02/23/2014 11:43 AM, Armin Rigo wrote: > I've checked in some new code in rawstorage.py on default: > raw_storage_{get,set}item_unaligned(). It's regular RPython code that > checks if the argument is aligned or not, and if not, it falls back to > a memcpy(). It should be relatively jit-friendly. Note that I also > changed the non-unaligned functions so that, when untranslated, they > complain if the address is unaligned, whatever the platform. > > Can you try to change some or all of the raw_storage_{get,set}item() > calls in micronumpy? I don't really know which ones are at a risk of > seeing unaligned accesses, and which ones not. It seems that all of > them are at risk: see the failure in > test_numarray.AppTestRecordDtype.test_string_record. > > > A bient?t, > > Armin. From arigo at tunes.org Thu Feb 27 15:54:02 2014 From: arigo at tunes.org (Armin Rigo) Date: Thu, 27 Feb 2014 15:54:02 +0100 Subject: [pypy-dev] slow-ish multithreaded primitives In-Reply-To: References: Message-ID: Hi Dima, On 25 February 2014 16:45, Dima Tisnek wrote: > Armin, is there really a semantical change? > Consider invocations valid in 2.7, (i.e. without timeout argument), is > it not the same then? It's different: Python 3.x acquire() can be interrupted by signals, whereas Python 2.x acquire() cannot. > should this code be in nightly builds? Yes. Armin From norbert.raimund.leisner at arcor.de Fri Feb 28 08:03:36 2014 From: norbert.raimund.leisner at arcor.de (norbert.raimund.leisner at arcor.de) Date: Fri, 28 Feb 2014 08:03:36 +0100 (CET) Subject: [pypy-dev] pypy 2.2.1 win32 Message-ID: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> Hello support-team, I have installed pypy 2.2.1 win32 for my OS Windows XP SP 3 -32 bit, Python 2.7, MSI Microsoft Visual C++ 2008 SP1 Redistributable Package (x86). Now my question: Must be Python 2.7 deinstalled and replaced by Pythonv2.7.6 or not? I ask you because a chess program "Sunfish" https://github.com/thomasahle/sunfish/ is using pypy. Best wishes, Norbert From fijall at gmail.com Fri Feb 28 08:39:20 2014 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 28 Feb 2014 09:39:20 +0200 Subject: [pypy-dev] pypy 2.2.1 win32 In-Reply-To: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> References: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> Message-ID: On Fri, Feb 28, 2014 at 9:03 AM, wrote: > Hello support-team, > > I have installed pypy 2.2.1 win32 for my OS Windows XP SP 3 -32 bit, Python 2.7, MSI Microsoft Visual C++ 2008 SP1 Redistributable Package (x86). > > Now my question: > Must be Python 2.7 deinstalled and replaced by Pythonv2.7.6 or not? As far as I understand your question, the answer is no. Various versions of Python (and PyPy) can happily coexist next to each other. Cheers, fijal From johan.rade at gmail.com Fri Feb 28 12:47:52 2014 From: johan.rade at gmail.com (=?ISO-8859-1?Q?Johan_R=E5de?=) Date: Fri, 28 Feb 2014 12:47:52 +0100 Subject: [pypy-dev] pypy 2.2.1 win32 In-Reply-To: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> References: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> Message-ID: On 2014-02-28 08:03, norbert.raimund.leisner at arcor.de wrote: > Hello support-team, > > I have installed pypy 2.2.1 win32 for my OS Windows XP SP 3 -32 bit, Python 2.7, MSI Microsoft Visual C++ 2008 SP1 Redistributable Package (x86). > > Now my question: > Must be Python 2.7 deinstalled and replaced by Pythonv2.7.6 or not? > > I ask you because a chess program "Sunfish" https://github.com/thomasahle/sunfish/ is using pypy. > > Best wishes, > Norbert > Hello Norbert, A complete chess engine in 111 lines! That's pretty cool. I just downloaded Sunfish and tried it with PyPy 2.2.1 on a Windows 7 computer. It worked fine. No problems. And I have at least four different Python versions installed. As Maciej wrote, they happily coexist. Is it not working for you? If not, can you please provide us with more information. Best wishes, Johan From cfbolz at gmx.de Fri Feb 28 13:05:48 2014 From: cfbolz at gmx.de (Carl Friedrich Bolz) Date: Fri, 28 Feb 2014 13:05:48 +0100 Subject: [pypy-dev] pypy 2.2.1 win32 In-Reply-To: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> References: <366629207.1251178.1393571016417.JavaMail.ngmail@webmail11.arcor-online.net> Message-ID: <53107B9C.4000705@gmx.de> Hi Norbert, On 28/02/14 08:03, norbert.raimund.leisner at arcor.de wrote: > I ask you because a chess program "Sunfish" https://github.com/thomasahle/sunfish/ is using pypy. Unrelated to your actual question, this sounds like a very cool addition to our benchmark set. Somebody feel like adding it? Cheers, Carl Friedrich From kostia.lopuhin at gmail.com Fri Feb 28 17:13:47 2014 From: kostia.lopuhin at gmail.com (=?KOI8-R?B?68/T1NEg7M/Q1cjJzg==?=) Date: Fri, 28 Feb 2014 20:13:47 +0400 Subject: [pypy-dev] Regression for lxml-cffi install (?) In-Reply-To: References: Message-ID: Maybe I am doing something entirely wrong here (e.g. the install command may be bogus). I would appreciate any advice :) 2014-02-28 20:11 GMT+04:00 ????? ??????? : > Hello! > https://bugs.pypy.org does not work at the moment, so posting here. > > pip install -e git+git://github.com/amauryfa/lxml.git at cffi#egg=lxml-cffi > > works under pypy 2.1 and fails with pypy nighly. > Not sure that this does really belong to pypy bugtracker, but it looks > like a regression. Tested under Ubuntu 12.04 and OS X, on the same > machines. In order to install lxml from source on ubuntu 12.04, I > installed the following ubuntu packages: libxml2 libxslt1-dev > zlib1g-dev > > pypy nighly (it failed on older nightlies too, and I can check on > other releases if needed) > > Python 2.7.3 (8df8a01902f8, Feb 27 2014, 23:00:44) > [PyPy 2.3.0-alpha0 with GCC 4.6.3] on linux2 > > Error under OS X (here error message is better than with gcc): > > > VerificationError: CompileError: command 'cc' failed with exit status 1 > > Complete output from command python setup.py egg_info: > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1555:18: > warning: incompatible pointer types initializing 'xmlBuffer **' (aka > 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka > 'struct _xmlBuf **') [-Wincompatible-pointer-types] > > > { xmlBuffer * *tmp = &p->buffer; (void)tmp; } > > > ^ ~~~~~~~~~~ > > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1556:18: > warning: incompatible pointer types initializing 'xmlBuffer **' (aka > 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka > 'struct _xmlBuf **') [-Wincompatible-pointer-types] > > > { xmlBuffer * *tmp = &p->conv; (void)tmp; } > > > ^ ~~~~~~~~ > > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: > error: cannot take the address of an rvalue of type 'const xmlChar *' > (aka 'const unsigned char *') > > > return (&XML_XML_NAMESPACE); > > > ^~~~~~~~~~~~~~~~~~ > > > 2 warnings and 1 error generated. > > > Building lxml version 3.2.1. > > > Building without Cython. > > > Using build configuration of libxslt 1.1.28 > > > Building against libxml2/libxslt in the following directory: /opt/local/lib > > > > > > Full error under Ubuntu 12.04: > > (testlxml_nighly)[DEV] kostia at dev ~ $ pip install -e > git+git://github.com/amauryfa/lxml.git at cffi#egg=lxml-cffi > > Obtaining lxml-cffi from > git+git://github.com/amauryfa/lxml.git at cffi#egg=lxml-cffi > > Updating ./testlxml_nighly/src/lxml-cffi clone (to cffi) > > Running setup.py > (path:/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py) egg_info > for package lxml-cffi > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In > function '_cffi_var_XML_XML_NAMESPACE': > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: > error: lvalue required as unary '&' operand > > Building lxml version 3.2.1. > > Building without Cython. > > Using build configuration of libxslt 1.1.26 > > Building against libxml2/libxslt in the following directory: > /usr/lib/x86_64-linux-gnu > > Traceback (most recent call last): > > File "app_main.py", line 72, in run_toplevel > > File "app_main.py", line 578, in run_it > > File "", line 17, in > > File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line > 220, in > > **setup_extra_options() > > File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line > 131, in setup_extra_options > > STATIC_CFLAGS, STATIC_BINARIES) > > File "setupinfo.py", line 161, in ext_modules > > import lxml.etree > > File "src/lxml/../lxml-cffi/etree.py", line 7, in > > from .apihelpers import _assertValidNode, _assertValidDoc > > File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in > > from .includes.etree_defs import _isString, _getNs, _isElement > > File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", > line 1, in > > from . import tree > > File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line > 490, in > > libraries=['xml2']) > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", > line 339, in verify > > lib = self.verifier.load_library() > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", > line 74, in load_library > > self._compile_module() > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", > line 139, in _compile_module > > outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", > line 25, in compile > > outputfilename = _build(tmpdir, ext) > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", > line 50, in _build > > raise VerificationError('%s: %s' % (e.__class__.__name__, e)) > > VerificationError: CompileError: command 'cc' failed with exit status 1 > > Complete output from command python setup.py egg_info: > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In > function '_cffi_var_XML_XML_NAMESPACE': > > > src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: > error: lvalue required as unary '&' operand > > > Building lxml version 3.2.1. > > > Building without Cython. > > > Using build configuration of libxslt 1.1.26 > > > Building against libxml2/libxslt in the following directory: > /usr/lib/x86_64-linux-gnu > > > Traceback (most recent call last): > > > File "app_main.py", line 72, in run_toplevel > > > File "app_main.py", line 578, in run_it > > > File "", line 17, in > > > File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line > 220, in > > > **setup_extra_options() > > > File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line > 131, in setup_extra_options > > > STATIC_CFLAGS, STATIC_BINARIES) > > > File "setupinfo.py", line 161, in ext_modules > > > import lxml.etree > > > File "src/lxml/../lxml-cffi/etree.py", line 7, in > > > from .apihelpers import _assertValidNode, _assertValidDoc > > > File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in > > > from .includes.etree_defs import _isString, _getNs, _isElement > > > File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", > line 1, in > > > from . import tree > > > File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, > in > > > libraries=['xml2']) > > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", > line 339, in verify > > > lib = self.verifier.load_library() > > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", > line 74, in load_library > > > self._compile_module() > > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", > line 139, in _compile_module > > > outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) > > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", > line 25, in compile > > > outputfilename = _build(tmpdir, ext) > > > File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", > line 50, in _build > > > raise VerificationError('%s: %s' % (e.__class__.__name__, e)) > VerificationError: CompileError: command 'cc' failed with exit status 1 > ---------------------------------------- > > Cleaning up... From kostia.lopuhin at gmail.com Fri Feb 28 17:11:31 2014 From: kostia.lopuhin at gmail.com (=?KOI8-R?B?68/T1NEg7M/Q1cjJzg==?=) Date: Fri, 28 Feb 2014 20:11:31 +0400 Subject: [pypy-dev] Regression for lxml-cffi install (?) Message-ID: Hello! https://bugs.pypy.org does not work at the moment, so posting here. pip install -e git+git://github.com/amauryfa/lxml.git at cffi#egg=lxml-cffi works under pypy 2.1 and fails with pypy nighly. Not sure that this does really belong to pypy bugtracker, but it looks like a regression. Tested under Ubuntu 12.04 and OS X, on the same machines. In order to install lxml from source on ubuntu 12.04, I installed the following ubuntu packages: libxml2 libxslt1-dev zlib1g-dev pypy nighly (it failed on older nightlies too, and I can check on other releases if needed) Python 2.7.3 (8df8a01902f8, Feb 27 2014, 23:00:44) [PyPy 2.3.0-alpha0 with GCC 4.6.3] on linux2 Error under OS X (here error message is better than with gcc): VerificationError: CompileError: command 'cc' failed with exit status 1 Complete output from command python setup.py egg_info: src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1555:18: warning: incompatible pointer types initializing 'xmlBuffer **' (aka 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka 'struct _xmlBuf **') [-Wincompatible-pointer-types] { xmlBuffer * *tmp = &p->buffer; (void)tmp; } ^ ~~~~~~~~~~ src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1556:18: warning: incompatible pointer types initializing 'xmlBuffer **' (aka 'struct _xmlBuffer **') with an expression of type 'xmlBufPtr *' (aka 'struct _xmlBuf **') [-Wincompatible-pointer-types] { xmlBuffer * *tmp = &p->conv; (void)tmp; } ^ ~~~~~~~~ src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: cannot take the address of an rvalue of type 'const xmlChar *' (aka 'const unsigned char *') return (&XML_XML_NAMESPACE); ^~~~~~~~~~~~~~~~~~ 2 warnings and 1 error generated. Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.28 Building against libxml2/libxslt in the following directory: /opt/local/lib Full error under Ubuntu 12.04: (testlxml_nighly)[DEV] kostia at dev ~ $ pip install -e git+git://github.com/amauryfa/lxml.git at cffi#egg=lxml-cffi Obtaining lxml-cffi from git+git://github.com/amauryfa/lxml.git at cffi#egg=lxml-cffi Updating ./testlxml_nighly/src/lxml-cffi clone (to cffi) Running setup.py (path:/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py) egg_info for package lxml-cffi src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In function '_cffi_var_XML_XML_NAMESPACE': src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: lvalue required as unary '&' operand Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "app_main.py", line 578, in run_it File "", line 17, in File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 220, in **setup_extra_options() File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 131, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "setupinfo.py", line 161, in ext_modules import lxml.etree File "src/lxml/../lxml-cffi/etree.py", line 7, in from .apihelpers import _assertValidNode, _assertValidDoc File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in from .includes.etree_defs import _isString, _getNs, _isElement File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", line 1, in from . import tree File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, in libraries=['xml2']) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", line 339, in verify lib = self.verifier.load_library() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 74, in load_library self._compile_module() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 139, in _compile_module outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 25, in compile outputfilename = _build(tmpdir, ext) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 50, in _build raise VerificationError('%s: %s' % (e.__class__.__name__, e)) VerificationError: CompileError: command 'cc' failed with exit status 1 Complete output from command python setup.py egg_info: src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c: In function '_cffi_var_XML_XML_NAMESPACE': src/lxml-cffi/includes/__pycache__/_cffi__gd9dd9435x1886a454.c:1603:11: error: lvalue required as unary '&' operand Building lxml version 3.2.1. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib/x86_64-linux-gnu Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "app_main.py", line 578, in run_it File "", line 17, in File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 220, in **setup_extra_options() File "/home/kostia/testlxml_nighly/src/lxml-cffi/setup.py", line 131, in setup_extra_options STATIC_CFLAGS, STATIC_BINARIES) File "setupinfo.py", line 161, in ext_modules import lxml.etree File "src/lxml/../lxml-cffi/etree.py", line 7, in from .apihelpers import _assertValidNode, _assertValidDoc File "src/lxml/../lxml-cffi/apihelpers.py", line 3, in from .includes.etree_defs import _isString, _getNs, _isElement File "src/lxml/includes/../../lxml-cffi/includes/etree_defs.py", line 1, in from . import tree File "src/lxml/includes/../../lxml-cffi/includes/tree.py", line 490, in libraries=['xml2']) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/api.py", line 339, in verify lib = self.verifier.load_library() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 74, in load_library self._compile_module() File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/verifier.py", line 139, in _compile_module outputfilename = ffiplatform.compile(tmpdir, self.get_extension()) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 25, in compile outputfilename = _build(tmpdir, ext) File "/home/chtd/pypy-c-jit-69524-8df8a01902f8-linux64/lib_pypy/cffi/ffiplatform.py", line 50, in _build raise VerificationError('%s: %s' % (e.__class__.__name__, e)) VerificationError: CompileError: command 'cc' failed with exit status 1 ---------------------------------------- Cleaning up...