[pypy-commit] pypy numpy-refactor: merge default
bdkearns
noreply at buildbot.pypy.org
Thu Feb 27 07:11:20 CET 2014
Author: Brian Kearns <bdkearns at gmail.com>
Branch: numpy-refactor
Changeset: r69491:8da9e6ab9957
Date: 2014-02-26 23:05 -0500
http://bitbucket.org/pypy/pypy/changeset/8da9e6ab9957/
Log: merge default
diff --git a/include/PyPy.h b/include/PyPy.h
--- a/include/PyPy.h
+++ b/include/PyPy.h
@@ -8,9 +8,14 @@
extern "C" {
#endif
+/* You should call this first once. */
+#define pypy_init(need_threads) do { pypy_asm_stack_bottom(); \
+rpython_startup_code();\
+ if (need_threads) pypy_init_threads(); } while (0)
-/* You should call this first once. */
+// deprecated interface
void rpython_startup_code(void);
+void pypy_init_threads(void);
/* Initialize the home directory of PyPy. It is necessary to call this.
@@ -26,11 +31,10 @@
/* If your program has multiple threads, then you need to call
- pypy_init_threads() once at init time, and then pypy_thread_attach()
- once in each other thread that just started and in which you want to
- run Python code (including via callbacks, see below).
+ pypy_thread_attach() once in each other thread that just started
+ and in which you want to run Python code (including via callbacks,
+ see below). DO NOT CALL IT IN THE MAIN THREAD
*/
-void pypy_init_threads(void);
void pypy_thread_attach(void);
diff --git a/pypy/doc/embedding.rst b/pypy/doc/embedding.rst
new file mode 100644
--- /dev/null
+++ b/pypy/doc/embedding.rst
@@ -0,0 +1,101 @@
+
+PyPy has a very minimal and a very strange embedding interface, based on
+the usage of `cffi`_ and the philosophy that Python is a better language in C.
+It was developed in collaboration with Roberto De Ioris from the `uwsgi`_
+project. The `PyPy uwsgi plugin`_ is a good example of usage of such interface.
+
+The first thing that you need, that we plan to change in the future, is to
+compile PyPy yourself with an option ``--shared``. Consult the
+`how to compile PyPy`_ doc for details. That should result in ``libpypy.so``
+or ``pypy.dll`` file or something similar, depending on your platform. Consult
+your platform specification for details.
+
+The resulting shared library has very few functions that are however enough
+to make a full API working, provided you'll follow a few principles. The API
+is:
+
+.. function:: void pypy_init(int need_threads);
+
+ This is a function that you have to call (once) before calling anything.
+ It initializes the RPython/PyPy GC and does a bunch of necessary startup
+ code. This function cannot fail. Pass 1 in case you need thread support, 0
+ otherwise.
+
+.. function:: long pypy_setup_home(char* home, int verbose);
+
+ This is another function that you have to call at some point, without
+ it you would not be able to find the standard library (and run pretty much
+ nothing). Arguments:
+
+ * ``home``: null terminated path
+
+ * ``verbose``: if non-zero, would print error messages to stderr
+
+ Function returns 0 on success or 1 on failure, can be called multiple times
+ until the library is found.
+
+.. function:: int pypy_execute_source(char* source);
+
+ Execute the source code given in the ``source`` argument. Will print
+ the error message to stderr upon failure and return 1, otherwise returns 0.
+ You should really do your own error handling in the source. It'll acquire
+ the GIL.
+
+.. function:: void pypy_thread_attach(void);
+
+ In case your application uses threads that are initialized outside of PyPy,
+ you need to call this function to tell the PyPy GC to track this thread.
+ Note that this function is not thread-safe itself, so you need to guard it
+ with a mutex. Do not call it from the main thread.
+
+Simple example
+--------------
+
+Note that this API is a lot more minimal than say CPython C API, so at first
+it's obvious to think that you can't do much. However, the trick is to do
+all the logic in Python and expose it via `cffi`_ callbacks. Let's assume
+we're on linux and pypy is put in ``/opt/pypy`` (a source checkout) and
+library is in ``/opt/pypy/libpypy-c.so``. We write a little C program
+(for simplicity assuming that all operations will be performed::
+
+ #include "include/PyPy.h"
+ #include <stdio.h>
+
+ const char source[] = "print 'hello from pypy'";
+
+ int main()
+ {
+ int res;
+
+ rpython_startup_code();
+ res = pypy_execute_source((char*)source);
+ if (res) {
+ printf("Error calling pypy_execute_source!\n");
+ }
+ return res;
+ }
+
+If we save it as ``x.c`` now, compile it and run it with::
+
+ fijal at hermann:/opt/pypy$ gcc -o x x.c -lpypy-c -L.
+ fijal at hermann:~/src/pypy$ LD_LIBRARY_PATH=. ./x
+ hello from pypy
+
+Worked!
+
+More advanced example
+---------------------
+
+Typically we need something more to do than simply execute source. The following
+is a fully fledged example, please consult cffi documentation for details.
+
+xxx
+
+Threading
+---------
+
+XXXX I don't understand what's going on, discuss with unbit
+
+.. _`cffi`: http://cffi.readthedocs.org/
+.. _`uwsgi`: http://uwsgi-docs.readthedocs.org/en/latest/
+.. _`PyPy uwsgi plugin`: http://uwsgi-docs.readthedocs.org/en/latest/PyPy.html
diff --git a/pypy/doc/getting-started.rst b/pypy/doc/getting-started.rst
--- a/pypy/doc/getting-started.rst
+++ b/pypy/doc/getting-started.rst
@@ -145,11 +145,13 @@
After you successfully manage to get PyPy's source you can read more about:
- `Building and using PyPy's Python interpreter`_
+ - `Embedding PyPy`_
- `Learning more about the RPython toolchain and how to develop (with) PyPy`_
- `Tutorial for how to write an interpreter with the RPython toolchain and make it fast`_
- `Look at our benchmark results`_
.. _`Building and using PyPy's Python interpreter`: getting-started-python.html
+.. _`Embedding PyPy`: embedding.html
.. _`Learning more about the RPython toolchain and how to develop (with) PyPy`: getting-started-dev.html
.. _`Tutorial for how to write an interpreter with the RPython toolchain and make it fast`: http://morepypy.blogspot.com/2011/04/tutorial-writing-interpreter-with-pypy.html
.. _`Look at our benchmark results`: http://speed.pypy.org
diff --git a/pypy/goal/targetpypystandalone.py b/pypy/goal/targetpypystandalone.py
--- a/pypy/goal/targetpypystandalone.py
+++ b/pypy/goal/targetpypystandalone.py
@@ -82,6 +82,7 @@
from rpython.rlib.entrypoint import entrypoint
from rpython.rtyper.lltypesystem import rffi, lltype
+ from rpython.rtyper.lltypesystem.lloperation import llop
w_pathsetter = space.appexec([], """():
def f(path):
@@ -93,6 +94,7 @@
@entrypoint('main', [rffi.CCHARP, rffi.INT], c_name='pypy_setup_home')
def pypy_setup_home(ll_home, verbose):
from pypy.module.sys.initpath import pypy_find_stdlib
+ llop.gc_stack_bottom(lltype.Void)
verbose = rffi.cast(lltype.Signed, verbose)
if ll_home:
home = rffi.charp2str(ll_home)
@@ -120,8 +122,11 @@
@entrypoint('main', [rffi.CCHARP], c_name='pypy_execute_source')
def pypy_execute_source(ll_source):
+ rffi.aroundstate.after()
+ llop.gc_stack_bottom(lltype.Void)
source = rffi.charp2str(ll_source)
res = _pypy_execute_source(source)
+ rffi.aroundstate.before()
return rffi.cast(rffi.INT, res)
@entrypoint('main', [], c_name='pypy_init_threads')
diff --git a/pypy/module/cpyext/ndarrayobject.py b/pypy/module/cpyext/ndarrayobject.py
--- a/pypy/module/cpyext/ndarrayobject.py
+++ b/pypy/module/cpyext/ndarrayobject.py
@@ -9,8 +9,7 @@
from pypy.module.cpyext.api import PyObject
from pypy.module.micronumpy.interp_numarray import W_NDimArray, array
from pypy.module.micronumpy.interp_dtype import get_dtype_cache, W_Dtype
-from pypy.module.micronumpy.arrayimpl.concrete import ConcreteArray
-from pypy.module.micronumpy.arrayimpl.scalar import Scalar
+from pypy.module.micronumpy.concrete import ConcreteArray
from rpython.rlib.rawstorage import RAW_STORAGE_PTR
NPY_C_CONTIGUOUS = 0x0001
@@ -167,7 +166,7 @@
# void *data = PyArray_DATA(arr);
impl = w_array.implementation
w_array = W_NDimArray.from_shape(space, [1], impl.dtype)
- w_array.implementation.setitem(0, impl.value)
+ w_array.implementation.setitem(0, impl.getitem(impl.start + 0))
w_array.implementation.shape = []
return w_array
@@ -214,12 +213,8 @@
order='C', owning=False, w_subtype=None):
shape, dtype = get_shape_and_dtype(space, nd, dims, typenum)
storage = rffi.cast(RAW_STORAGE_PTR, data)
- if nd == 0:
- w_val = dtype.itemtype.box_raw_data(storage)
- return W_NDimArray(Scalar(dtype, w_val))
- else:
- return W_NDimArray.from_shape_and_storage(space, shape, storage, dtype,
- order=order, owning=owning, w_subtype=w_subtype)
+ return W_NDimArray.from_shape_and_storage(space, shape, storage, dtype,
+ order=order, owning=owning, w_subtype=w_subtype)
@cpython_api([Py_ssize_t, rffi.LONGP, Py_ssize_t], PyObject)
diff --git a/pypy/module/cpyext/test/test_ndarrayobject.py b/pypy/module/cpyext/test/test_ndarrayobject.py
--- a/pypy/module/cpyext/test/test_ndarrayobject.py
+++ b/pypy/module/cpyext/test/test_ndarrayobject.py
@@ -77,7 +77,7 @@
def test_FromAny_scalar(self, space, api):
a0 = scalar(space)
- assert a0.implementation.get_scalar_value().value == 10.
+ assert a0.get_scalar_value().value == 10.
a = api._PyArray_FromAny(a0, NULL, 0, 0, 0, NULL)
assert api._PyArray_NDIM(a) == 0
diff --git a/pypy/module/micronumpy/base.py b/pypy/module/micronumpy/base.py
--- a/pypy/module/micronumpy/base.py
+++ b/pypy/module/micronumpy/base.py
@@ -31,7 +31,6 @@
@staticmethod
def from_shape(space, shape, dtype, order='C', w_instance=None):
from pypy.module.micronumpy import concrete
-
strides, backstrides = calc_strides(shape, dtype.base, order)
impl = concrete.ConcreteArray(shape, dtype.base, order, strides,
backstrides)
@@ -43,7 +42,6 @@
def from_shape_and_storage(space, shape, storage, dtype, order='C', owning=False,
w_subtype=None, w_base=None, writable=True):
from pypy.module.micronumpy import concrete
- assert shape
strides, backstrides = calc_strides(shape, dtype, order)
if w_base is not None:
if owning:
@@ -56,7 +54,6 @@
impl = concrete.ConcreteNonWritableArrayWithBase(shape, dtype, order,
strides, backstrides,
storage, w_base)
-
elif owning:
# Will free storage when GCd
impl = concrete.ConcreteArray(shape, dtype, order, strides,
diff --git a/rpython/jit/codewriter/jtransform.py b/rpython/jit/codewriter/jtransform.py
--- a/rpython/jit/codewriter/jtransform.py
+++ b/rpython/jit/codewriter/jtransform.py
@@ -499,6 +499,16 @@
def rewrite_op_hint(self, op):
hints = op.args[1].value
+
+ # hack: if there are both 'promote' and 'promote_string', kill
+ # one of them based on the type of the value
+ if hints.get('promote_string') and hints.get('promote'):
+ hints = hints.copy()
+ if op.args[0].concretetype == lltype.Ptr(rstr.STR):
+ del hints['promote']
+ else:
+ del hints['promote_string']
+
if hints.get('promote') and op.args[0].concretetype is not lltype.Void:
assert op.args[0].concretetype != lltype.Ptr(rstr.STR)
kind = getkind(op.args[0].concretetype)
diff --git a/rpython/jit/codewriter/test/test_jtransform.py b/rpython/jit/codewriter/test/test_jtransform.py
--- a/rpython/jit/codewriter/test/test_jtransform.py
+++ b/rpython/jit/codewriter/test/test_jtransform.py
@@ -1050,6 +1050,37 @@
assert op1.result == v2
assert op0.opname == '-live-'
+def test_double_promote_str():
+ PSTR = lltype.Ptr(rstr.STR)
+ v1 = varoftype(PSTR)
+ v2 = varoftype(PSTR)
+ tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+ op1 = SpaceOperation('hint',
+ [v1, Constant({'promote_string': True}, lltype.Void)],
+ v2)
+ op2 = SpaceOperation('hint',
+ [v1, Constant({'promote_string': True,
+ 'promote': True}, lltype.Void)],
+ v2)
+ lst1 = tr.rewrite_operation(op1)
+ lst2 = tr.rewrite_operation(op2)
+ assert lst1 == lst2
+
+def test_double_promote_nonstr():
+ v1 = varoftype(lltype.Signed)
+ v2 = varoftype(lltype.Signed)
+ tr = Transformer(FakeCPU(), FakeBuiltinCallControl())
+ op1 = SpaceOperation('hint',
+ [v1, Constant({'promote': True}, lltype.Void)],
+ v2)
+ op2 = SpaceOperation('hint',
+ [v1, Constant({'promote_string': True,
+ 'promote': True}, lltype.Void)],
+ v2)
+ lst1 = tr.rewrite_operation(op1)
+ lst2 = tr.rewrite_operation(op2)
+ assert lst1 == lst2
+
def test_unicode_concat():
# test that the oopspec is present and correctly transformed
PSTR = lltype.Ptr(rstr.UNICODE)
diff --git a/rpython/jit/metainterp/test/test_list.py b/rpython/jit/metainterp/test/test_list.py
--- a/rpython/jit/metainterp/test/test_list.py
+++ b/rpython/jit/metainterp/test/test_list.py
@@ -98,8 +98,8 @@
self.check_resops(setarrayitem_gc=0, call=0, getarrayitem_gc=0)
def test_vlist_alloc_and_set(self):
- # the check_loops fails, because [non-null] * n is not supported yet
- # (it is implemented as a residual call)
+ # the check_loops fails, because [non-null] * n is only supported
+ # if n < 15 (otherwise it is implemented as a residual call)
jitdriver = JitDriver(greens = [], reds = ['n'])
def f(n):
l = [1] * 20
@@ -116,7 +116,7 @@
res = self.meta_interp(f, [10], listops=True)
assert res == f(10)
- py.test.skip("'[non-null] * n' gives a residual call so far")
+ py.test.skip("'[non-null] * n' for n >= 15 gives a residual call so far")
self.check_loops(setarrayitem_gc=0, getarrayitem_gc=0, call=0)
def test_arraycopy_simpleoptimize(self):
@@ -287,6 +287,74 @@
assert res == 5
self.check_resops(call=0)
+ def test_list_mul_virtual(self):
+ class Foo:
+ def __init__(self, l):
+ self.l = l
+ l[0] = self
+
+ myjitdriver = JitDriver(greens = [], reds = ['y'])
+ def f(y):
+ while y > 0:
+ myjitdriver.jit_merge_point(y=y)
+ Foo([None] * 5)
+ y -= 1
+ return 42
+
+ self.meta_interp(f, [5])
+ self.check_resops({'int_sub': 2,
+ 'int_gt': 2,
+ 'guard_true': 2,
+ 'jump': 1})
+
+ def test_list_mul_virtual_nonzero(self):
+ class base:
+ pass
+ class Foo(base):
+ def __init__(self, l):
+ self.l = l
+ l[0] = self
+ class nil(base):
+ pass
+
+ nil = nil()
+
+ myjitdriver = JitDriver(greens = [], reds = ['y'])
+ def f(y):
+ while y > 0:
+ myjitdriver.jit_merge_point(y=y)
+ Foo([nil] * 5)
+ y -= 1
+ return 42
+
+ self.meta_interp(f, [5])
+ self.check_resops({'int_sub': 2,
+ 'int_gt': 2,
+ 'guard_true': 2,
+ 'jump': 1})
+
+ def test_list_mul_unsigned_virtual(self):
+ from rpython.rlib.rarithmetic import r_uint
+
+ class Foo:
+ def __init__(self, l):
+ self.l = l
+ l[0] = self
+
+ myjitdriver = JitDriver(greens = [], reds = ['y'])
+ def f(y):
+ while y > 0:
+ myjitdriver.jit_merge_point(y=y)
+ Foo([None] * r_uint(5))
+ y -= 1
+ return 42
+
+ self.meta_interp(f, [5])
+ self.check_resops({'int_sub': 2,
+ 'int_gt': 2,
+ 'guard_true': 2,
+ 'jump': 1})
+
class TestLLtype(ListTests, LLJitMixin):
def test_listops_dont_invalidate_caches(self):
diff --git a/rpython/rlib/jit.py b/rpython/rlib/jit.py
--- a/rpython/rlib/jit.py
+++ b/rpython/rlib/jit.py
@@ -130,7 +130,9 @@
if promote_args != 'all':
args = [args[int(i)] for i in promote_args.split(",")]
for arg in args:
- code.append(" %s = hint(%s, promote=True)\n" % (arg, arg))
+ code.append( #use both hints, and let jtransform pick the right one
+ " %s = hint(%s, promote=True, promote_string=True)\n" %
+ (arg, arg))
code.append(" return _orig_func_unlikely_name(%s)\n" % (argstring, ))
d = {"_orig_func_unlikely_name": func, "hint": hint}
exec py.code.Source("\n".join(code)).compile() in d
diff --git a/rpython/rtyper/test/test_generator.py b/rpython/rtyper/test/test_generator.py
--- a/rpython/rtyper/test/test_generator.py
+++ b/rpython/rtyper/test/test_generator.py
@@ -88,3 +88,16 @@
return s
res = self.interpret(g, [])
assert res == 6
+
+ def test_send(self):
+ def f():
+ yield (yield 1) + 1
+ def g():
+ gen = f()
+ res = f.send(2)
+ assert res == 1
+ res = f.next()
+ assert res == 3
+
+ res = self.interpret(g, [])
+
diff --git a/rpython/rtyper/test/test_rlist.py b/rpython/rtyper/test/test_rlist.py
--- a/rpython/rtyper/test/test_rlist.py
+++ b/rpython/rtyper/test/test_rlist.py
@@ -1619,3 +1619,17 @@
rgc.ll_arraycopy = old_arraycopy
#
assert 2 <= res <= 10
+
+ def test_alloc_and_set(self):
+ def fn(i):
+ lst = [0] * r_uint(i)
+ return lst
+ t, rtyper, graph = self.gengraph(fn, [int])
+ block = graph.startblock
+ seen = 0
+ for op in block.operations:
+ if op.opname in ['cast_int_to_uint', 'cast_uint_to_int']:
+ continue
+ assert op.opname == 'direct_call'
+ seen += 1
+ assert seen == 1
diff --git a/rpython/translator/transform.py b/rpython/translator/transform.py
--- a/rpython/translator/transform.py
+++ b/rpython/translator/transform.py
@@ -30,7 +30,7 @@
# [a] * b
# -->
# c = newlist(a)
-# d = mul(c, int b)
+# d = mul(c, b)
# -->
# d = alloc_and_set(b, a)
@@ -44,8 +44,7 @@
len(op.args) == 1):
length1_lists[op.result] = op.args[0]
elif (op.opname == 'mul' and
- op.args[0] in length1_lists and
- self.gettype(op.args[1]) is int):
+ op.args[0] in length1_lists):
new_op = SpaceOperation('alloc_and_set',
(op.args[1], length1_lists[op.args[0]]),
op.result)
More information about the pypy-commit
mailing list