[pypy-commit] pypy default: coalesced the 3 existing contributing docs into a single doc
mwjackson
pypy.commits at gmail.com
Wed May 9 13:29:18 EDT 2018
Author: Matt Jackson <email at mwjackson.net>
Branch:
Changeset: r94508:b84ccf3ff0d6
Date: 2018-04-21 19:30 +0100
http://bitbucket.org/pypy/pypy/changeset/b84ccf3ff0d6/
Log: coalesced the 3 existing contributing docs into a single doc
diff --git a/pypy/doc/contributing.rst b/pypy/doc/contributing.rst
new file mode 100644
--- /dev/null
+++ b/pypy/doc/contributing.rst
@@ -0,0 +1,490 @@
+Contributing Guidelines
+===========================
+
+.. contents::
+
+PyPy is a very large project that has a reputation of being hard to dive into.
+Some of this fame is warranted, some of it is purely accidental. There are three
+important lessons that everyone willing to contribute should learn:
+
+* PyPy has layers. There are many pieces of architecture that are very well
+ separated from each other. More about this below, but often the manifestation
+ of this is that things are at a different layer than you would expect them
+ to be. For example if you are looking for the JIT implementation, you will
+ not find it in the implementation of the Python programming language.
+
+* Because of the above, we are very serious about Test Driven Development.
+ It's not only what we believe in, but also that PyPy's architecture is
+ working very well with TDD in mind and not so well without it. Often
+ development means progressing in an unrelated corner, one unittest
+ at a time; and then flipping a giant switch, bringing it all together.
+ (It generally works out of the box. If it doesn't, then we didn't
+ write enough unit tests.) It's worth repeating - PyPy's
+ approach is great if you do TDD, and not so great otherwise.
+
+* PyPy uses an entirely different set of tools - most of them included
+ in the PyPy repository. There is no Makefile, nor autoconf. More below.
+
+The first thing to remember is that PyPy project is very different than most projects out there.
+It's also different from a classic compiler project, so academic courses
+about compilers often don't apply or lead in the wrong direction.
+
+Getting involved
+----------------
+
+PyPy employs an open development process. You are invited to join our
+`pypy-dev mailing list`_, our IRC channel (#pypy on freenode) or look at the
+other :ref:`contact possibilities <contact>`. Usually we give out commit
+rights fairly liberally, so if you want to do something with PyPy, you can
+become a committer. We also run frequent coding sprints which are separately
+announced and often happen around Python conferences such as EuroPython or
+PyCon. Upcoming events are usually announced on `the blog`_.
+
+.. _the blog: http://morepypy.blogspot.com
+.. _pypy-dev mailing list: http://mail.python.org/mailman/listinfo/pypy-dev
+
+
+Hacking
+--------
+
+The first and most important rule how _not_ to contribute to PyPy is
+"just hacking". This won't work. There are two major reasons why not
+-- build times are large and PyPy has very thick layer separation which
+make it harder to "just hack a feature". Instead, reach out on the dev mailing
+list or the IRC channel, and we're more than happy to help! :)
+
+Source Control
+==============
+
+Using Mercurial
+---------------
+
+PyPy development is based on Mercurial (hg). If you are not used to
+version control, the cycle for a new PyPy contributor goes typically
+like this:
+
+* Make an account on bitbucket_.
+
+* Go to https://bitbucket.org/pypy/pypy/ and click "fork" (left
+ icons). You get a fork of the repository, e.g. in
+ https://bitbucket.org/yourname/pypy/.
+
+* Clone this new repo (i.e. the fork) to your local machine with the command
+ ``hg clone ssh://hg@bitbucket.org/yourname/pypy``. It is a very slow
+ operation but only ever needs to be done once. See also
+ http://pypy.org/download.html#building-from-source .
+ If you already cloned
+ ``https://bitbucket.org/pypy/pypy`` before, even if some time ago,
+ then you can reuse the same clone by editing the file ``.hg/hgrc`` in
+ your clone to contain the line ``default =
+ ssh://hg@bitbucket.org/yourname/pypy``, and then do ``hg pull && hg
+ up``. If you already have such a clone but don't want to change it,
+ you can clone that copy with ``hg clone /path/to/other/copy``, and
+ then edit ``.hg/hgrc`` as above and do ``hg pull && hg up``.
+
+* Now you have a complete copy of the PyPy repo. Make a branch
+ with a command like ``hg branch name_of_your_branch``.
+
+* Edit things. Use ``hg diff`` to see what you changed. Use ``hg add``
+ to make Mercurial aware of new files you added, e.g. new test files.
+ Use ``hg status`` to see if there are such files. Write and run tests!
+ (See the rest of this page.)
+
+* Commit regularly with ``hg commit``. A one-line commit message is
+ fine. We love to have tons of commits; make one as soon as you have
+ some progress, even if it is only some new test that doesn't pass yet,
+ or fixing things even if not all tests pass. Step by step, you are
+ building the history of your changes, which is the point of a version
+ control system. (There are commands like ``hg log`` and ``hg up``
+ that you should read about later, to learn how to navigate this
+ history.)
+
+* The commits stay on your machine until you do ``hg push`` to "push"
+ them back to the repo named in the file ``.hg/hgrc``. Repos are
+ basically just collections of commits (a commit is also called a
+ changeset): there is one repo per url, plus one for each local copy on
+ each local machine. The commands ``hg push`` and ``hg pull`` copy
+ commits around, with the goal that all repos in question end up with
+ the exact same set of commits. By opposition, ``hg up`` only updates
+ the "working copy" by reading the local repository, i.e. it makes the
+ files that you see correspond to the latest (or any other) commit
+ locally present.
+
+* You should push often; there is no real reason not to. Remember that
+ even if they are pushed, with the setup above, the commits are (1)
+ only in ``bitbucket.org/yourname/pypy``, and (2) in the branch you
+ named. Yes, they are publicly visible, but don't worry about someone
+ walking around the thousands of repos on bitbucket saying "hah, look
+ at the bad coding style of that guy". Try to get into the mindset
+ that your work is not secret and it's fine that way. We might not
+ accept it as is for PyPy, asking you instead to improve some things,
+ but we are not going to judge you.
+
+* The final step is to open a pull request, so that we know that you'd
+ like to merge that branch back to the original ``pypy/pypy`` repo.
+ This can also be done several times if you have interesting
+ intermediate states, but if you get there, then we're likely to
+ proceed to the next stage, which is...
+
+* Get a regular account for pushing directly to
+ ``bitbucket.org/pypy/pypy`` (just ask and you'll get it, basically).
+ Once you have it you can rewrite your file ``.hg/hgrc`` to contain
+ ``default = ssh://hg@bitbucket.org/pypy/pypy``. Your changes will
+ then be pushed directly to the official repo, but (if you follow these
+ rules) they are still on a branch, and we can still review the
+ branches you want to merge.
+
+* If you get closer to the regular day-to-day development, you'll notice
+ that we generally push small changes as one or a few commits directly
+ to the branch ``default``. Also, we often collaborate even if we are
+ on other branches, which do not really "belong" to anyone. At this
+ point you'll need ``hg merge`` and learn how to resolve conflicts that
+ sometimes occur when two people try to push different commits in
+ parallel on the same branch. But it is likely an issue for later ``:-)``
+
+.. _bitbucket: https://bitbucket.org/
+
+
+Architecture
+============
+
+PyPy has layers. The 100 miles view:
+
+* :ref:`RPython <rpython:language>` is the language in which we write interpreters. Not the entire
+ PyPy project is written in RPython, only the parts that are compiled in
+ the translation process. The interesting point is that RPython has no parser,
+ it's compiled from the live python objects, which makes it possible to do
+ all kinds of metaprogramming during import time. In short, Python is a meta
+ programming language for RPython.
+
+ The RPython standard library is to be found in the ``rlib`` subdirectory.
+
+* The translation toolchain - this is the part that takes care of translating
+ RPython to flow graphs and then to C. There is more in the :doc:`architecture <architecture>`
+ document written about it.
+
+ It lives in the ``rpython`` directory: ``flowspace``, ``annotator``
+ and ``rtyper``.
+
+* Python Interpreter and modules
+
+ This is in the ``pypy`` directory. ``pypy/interpreter`` is a standard
+ interpreter for Python written in RPython. The fact that it is
+ RPython is not apparent at first. Built-in modules are written in
+ ``pypy/module/*``. Some modules that CPython implements in C are
+ simply written in pure Python; they are in the top-level ``lib_pypy``
+ directory. The standard library of Python (with a few changes to
+ accomodate PyPy) is in ``lib-python``.
+
+* :ref:`Just-in-Time Compiler (JIT) <rpython:jit>`: we have a tracing JIT that traces the
+ interpreter written in RPython, rather than the user program that it
+ interprets. As a result it applies to any interpreter, i.e. any
+ language. But getting it to work correctly is not trivial: it
+ requires a small number of precise "hints" and possibly some small
+ refactorings of the interpreter. The JIT itself also has several
+ almost-independent parts: the tracer itself in ``rpython/jit/metainterp``, the
+ optimizer in ``rpython/jit/metainterp/optimizer`` that optimizes a list of
+ residual operations, and the backend in ``rpython/jit/backend/<machine-name>``
+ that turns it into machine code. Writing a new backend is a
+ traditional way to get into the project.
+
+* Garbage Collectors (GC): as you may notice if you are used to CPython's
+ C code, there are no ``Py_INCREF/Py_DECREF`` equivalents in RPython code.
+ :ref:`rpython:garbage-collection` is inserted
+ during translation. Moreover, this is not reference counting; it is a real
+ GC written as more RPython code. The best one we have so far is in
+ ``rpython/memory/gc/incminimark.py``.
+
+Layers
+------
+
+PyPy has layers. Just like Ogres or onions.
+Those layers help us keep the respective parts separated enough
+to be worked on independently and make the complexity manageable. This is,
+again, just a sanity requirement for such a complex project. For example writing
+a new optimization for the JIT usually does **not** involve touching a Python
+interpreter at all or the JIT assembler backend or the garbage collector.
+Instead it requires writing small tests in
+``rpython/jit/metainterp/optimizeopt/test/test_*`` and fixing files there.
+After that, you can just compile PyPy and things should just work.
+
+The short list of layers for further reading. For each of those layers, a good
+entry point is a test subdirectory in respective directories. It usually
+describes (better or worse) the interfaces between the submodules. For the
+``pypy`` subdirectory, most tests are small snippets of python programs that
+check for correctness (calls ``AppTestXxx``) that will call the appropriate
+part of the interpreter. For the ``rpython`` directory, most tests are small
+RPython interpreters that perform certain tasks. To see how they translate
+to low-level graphs, run them with ``--view``. To see small interpreters
+with a JIT compiler, use ``--viewloops`` option.
+
+* **python interpreter** - it's the part implemented in the ``pypy/`` directory.
+ It's implemented in RPython, which is a high level static language with
+ classes, garbage collection, just-in-time compiler generation and the ability
+ to call C. A cool part about it is that it can be run untranslated, so all
+ the tests are runnable without translating PyPy.
+
+ **interpreter** contains the interpreter core
+
+ **objspace** contains implementations of various objects exported to
+ the Python layer
+
+ **module** directory contains extension modules written in RPython
+
+* **rpython compiler** that resides in ``rpython/annotator`` and
+ ``rpython/rtyper`` directories. Consult `Getting Started with RPython`_
+ for further reading
+
+* **JIT generator** lives in ``rpython/jit`` directory. optimizations live
+ in ``rpython/jit/metainterp/optimizeopt``, the main JIT in
+ ``rpython/jit/metainterp`` (runtime part) and
+ ``rpython/jit/codewriter`` (translation-time part). Backends live in
+ ``rpython/jit/backend``.
+
+* **garbage collection** lives in ``rpython/memory``
+
+The rest of directories serve specific niche goal and are unlikely a good
+entry point.
+
+.. _`Getting started with RPython`: http://rpython.readthedocs.org/en/latest/getting-started.html
+
+Where to start reading the sources
+----------------------------------
+
+PyPy is made from parts that are relatively independent of each other.
+You should start looking at the part that attracts you most (all paths are
+relative to the PyPy top level directory). You may look at our :doc:`directory reference <dir-reference>`
+or start off at one of the following points:
+
+* :source:`pypy/interpreter` contains the bytecode interpreter: bytecode dispatcher
+ in :source:`pypy/interpreter/pyopcode.py`, frame and code objects in
+ :source:`pypy/interpreter/eval.py` and :source:`pypy/interpreter/pyframe.py`,
+ function objects and argument passing in :source:`pypy/interpreter/function.py`
+ and :source:`pypy/interpreter/argument.py`, the object space interface
+ definition in :source:`pypy/interpreter/baseobjspace.py`, modules in
+ :source:`pypy/interpreter/module.py` and :source:`pypy/interpreter/mixedmodule.py`.
+ Core types supporting the bytecode interpreter are defined in :source:`pypy/interpreter/typedef.py`.
+
+* :source:`pypy/interpreter/pyparser` contains a recursive descent parser,
+ and grammar files that allow it to parse the syntax of various Python
+ versions. Once the grammar has been processed, the parser can be
+ translated by the above machinery into efficient code.
+
+* :source:`pypy/interpreter/astcompiler` contains the compiler. This
+ contains a modified version of the compiler package from CPython
+ that fixes some bugs and is translatable.
+
+* :source:`pypy/objspace/std` contains the :ref:`Standard object space <standard-object-space>`. The main file
+ is :source:`pypy/objspace/std/objspace.py`. For each type, the file
+ ``xxxobject.py`` contains the implementation for objects of type ``xxx``,
+ as a first approximation. (Some types have multiple implementations.)
+
+Testing
+=======
+
+Test driven development
+-----------------------
+
+Instead, we practice a lot of test driven development. This is partly because
+of very high quality requirements for compilers and partly because there is
+simply no other way to get around such complex project, that will keep you sane.
+There are probably people out there who are smart enough not to need it, we're
+not one of those. You may consider familiarizing yourself with `pytest`_,
+since this is a tool we use for tests.
+This leads to the next issue:
+
+.. _pytest: http://pytest.org/
+
+py.test and the py lib
+----------------------
+
+The `py.test testing tool`_ drives all our testing needs.
+
+We use the `py library`_ for filesystem path manipulations, terminal
+writing, logging and some other support functionality.
+
+You don't necessarily need to install these two libraries because
+we also ship them inlined in the PyPy source tree.
+
+.. _py library: http://pylib.readthedocs.org/
+
+Running PyPy's unit tests
+-------------------------
+
+PyPy development always was and is still thoroughly test-driven.
+We use the flexible `py.test testing tool`_ which you can `install independently
+<http://pytest.org/latest/getting-started.html#getstarted>`_ and use for other projects.
+
+The PyPy source tree comes with an inlined version of ``py.test``
+which you can invoke by typing::
+
+ python pytest.py -h
+
+This is usually equivalent to using an installed version::
+
+ py.test -h
+
+If you encounter problems with the installed version
+make sure you have the correct version installed which
+you can find out with the ``--version`` switch.
+
+You will need the `build requirements`_ to run tests successfully, since many of
+them compile little pieces of PyPy and then run the tests inside that minimal
+interpreter
+
+Now on to running some tests. PyPy has many different test directories
+and you can use shell completion to point at directories or files::
+
+ py.test pypy/interpreter/test/test_pyframe.py
+
+ # or for running tests of a whole subdirectory
+ py.test pypy/interpreter/
+
+See `py.test usage and invocations`_ for some more generic info
+on how you can run tests.
+
+Beware trying to run "all" pypy tests by pointing to the root
+directory or even the top level subdirectory ``pypy``. It takes
+hours and uses huge amounts of RAM and is not recommended.
+
+To run CPython regression tests you can point to the ``lib-python``
+directory::
+
+ py.test lib-python/2.7/test/test_datetime.py
+
+This will usually take a long time because this will run
+the PyPy Python interpreter on top of CPython. On the plus
+side, it's usually still faster than doing a full translation
+and running the regression test with the translated PyPy Python
+interpreter.
+
+.. _py.test testing tool: http://pytest.org
+.. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
+.. _`build requirements`: build.html#install-build-time-dependencies
+
+Tooling & Utilities
+===================
+
+If you are interested in the inner workings of the PyPy Python interpreter,
+there are some features of the untranslated Python interpreter that allow you
+to introspect its internals.
+
+
+Interpreter-level console
+-------------------------
+
+To start interpreting Python with PyPy, install a C compiler that is
+supported by distutils and use Python 2.7 or greater to run PyPy::
+
+ cd pypy
+ python bin/pyinteractive.py
+
+After a few seconds (remember: this is running on top of CPython), you should
+be at the PyPy prompt, which is the same as the Python prompt, but with an
+extra ">".
+
+If you press
+<Ctrl-C> on the console you enter the interpreter-level console, a
+usual CPython console. You can then access internal objects of PyPy
+(e.g. the :ref:`object space <objspace>`) and any variables you have created on the PyPy
+prompt with the prefix ``w_``::
+
+ >>>> a = 123
+ >>>> <Ctrl-C>
+ *** Entering interpreter-level console ***
+ >>> w_a
+ W_IntObject(123)
+
+The mechanism works in both directions. If you define a variable with the ``w_`` prefix on the interpreter-level, you will see it on the app-level::
+
+ >>> w_l = space.newlist([space.wrap(1), space.wrap("abc")])
+ >>> <Ctrl-D>
+ *** Leaving interpreter-level console ***
+
+ KeyboardInterrupt
+ >>>> l
+ [1, 'abc']
+
+Note that the prompt of the interpreter-level console is only '>>>' since
+it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under
+Linux) or <Ctrl-Z>, <Enter> (under Windows).
+
+Also note that not all modules are available by default in this mode (for
+example: ``_continuation`` needed by ``greenlet``) , you may need to use one of
+``--withmod-...`` command line options.
+
+You may be interested in reading more about the distinction between
+:ref:`interpreter-level and app-level <interpreter-level>`.
+
+pyinteractive.py options
+------------------------
+
+To list the PyPy interpreter command line options, type::
+
+ cd pypy
+ python bin/pyinteractive.py --help
+
+pyinteractive.py supports most of the options that CPython supports too (in addition to a
+large amount of options that can be used to customize pyinteractive.py).
+As an example of using PyPy from the command line, you could type::
+
+ python pyinteractive.py --withmod-time -c "from test import pystone; pystone.main(10)"
+
+Alternatively, as with regular Python, you can simply give a
+script name on the command line::
+
+ python pyinteractive.py --withmod-time ../../lib-python/2.7/test/pystone.py 10
+
+The ``--withmod-xxx`` option enables the built-in module ``xxx``. By
+default almost none of them are, because initializing them takes time.
+If you want anyway to enable all built-in modules, you can use
+``--allworkingmodules``.
+
+See our :doc:`configuration sections <config/index>` for details about what all the commandline
+options do.
+
+
+.. _trace example:
+
+Tracing bytecode and operations on objects
+------------------------------------------
+
+You can use a simple tracing mode to monitor the interpretation of
+bytecodes. To enable it, set ``__pytrace__ = 1`` on the interactive
+PyPy console::
+
+ >>>> __pytrace__ = 1
+ Tracing enabled
+ >>>> x = 5
+ <module>: LOAD_CONST 0 (5)
+ <module>: STORE_NAME 0 (x)
+ <module>: LOAD_CONST 1 (None)
+ <module>: RETURN_VALUE 0
+ >>>> x
+ <module>: LOAD_NAME 0 (x)
+ <module>: PRINT_EXPR 0
+ 5
+ <module>: LOAD_CONST 0 (None)
+ <module>: RETURN_VALUE 0
+ >>>>
+
+
+Demos
+-----
+
+The `example-interpreter`_ repository contains an example interpreter
+written using the RPython translation toolchain.
+
+.. _example-interpreter: https://bitbucket.org/pypy/example-interpreter
+
+
+graphviz & pygame for flow graph viewing (highly recommended)
+-------------------------------------------------------------
+
+graphviz and pygame are both necessary if you want to look at generated flow
+graphs:
+
+ graphviz: http://www.graphviz.org/Download.php
+
+ pygame: http://www.pygame.org/download.shtml
+
diff --git a/pypy/doc/getting-started-dev.rst b/pypy/doc/getting-started-dev.rst
deleted file mode 100644
--- a/pypy/doc/getting-started-dev.rst
+++ /dev/null
@@ -1,345 +0,0 @@
-Getting Started Developing With PyPy
-====================================
-
-.. contents::
-
-
-Using Mercurial
----------------
-
-PyPy development is based on Mercurial (hg). If you are not used to
-version control, the cycle for a new PyPy contributor goes typically
-like this:
-
-* Make an account on bitbucket_.
-
-* Go to https://bitbucket.org/pypy/pypy/ and click "fork" (left
- icons). You get a fork of the repository, e.g. in
- https://bitbucket.org/yourname/pypy/.
-
-* Clone this new repo (i.e. the fork) to your local machine with the command
- ``hg clone ssh://hg@bitbucket.org/yourname/pypy``. It is a very slow
- operation but only ever needs to be done once. See also
- http://pypy.org/download.html#building-from-source .
- If you already cloned
- ``https://bitbucket.org/pypy/pypy`` before, even if some time ago,
- then you can reuse the same clone by editing the file ``.hg/hgrc`` in
- your clone to contain the line ``default =
- ssh://hg@bitbucket.org/yourname/pypy``, and then do ``hg pull && hg
- up``. If you already have such a clone but don't want to change it,
- you can clone that copy with ``hg clone /path/to/other/copy``, and
- then edit ``.hg/hgrc`` as above and do ``hg pull && hg up``.
-
-* Now you have a complete copy of the PyPy repo. Make a branch
- with a command like ``hg branch name_of_your_branch``.
-
-* Edit things. Use ``hg diff`` to see what you changed. Use ``hg add``
- to make Mercurial aware of new files you added, e.g. new test files.
- Use ``hg status`` to see if there are such files. Write and run tests!
- (See the rest of this page.)
-
-* Commit regularly with ``hg commit``. A one-line commit message is
- fine. We love to have tons of commits; make one as soon as you have
- some progress, even if it is only some new test that doesn't pass yet,
- or fixing things even if not all tests pass. Step by step, you are
- building the history of your changes, which is the point of a version
- control system. (There are commands like ``hg log`` and ``hg up``
- that you should read about later, to learn how to navigate this
- history.)
-
-* The commits stay on your machine until you do ``hg push`` to "push"
- them back to the repo named in the file ``.hg/hgrc``. Repos are
- basically just collections of commits (a commit is also called a
- changeset): there is one repo per url, plus one for each local copy on
- each local machine. The commands ``hg push`` and ``hg pull`` copy
- commits around, with the goal that all repos in question end up with
- the exact same set of commits. By opposition, ``hg up`` only updates
- the "working copy" by reading the local repository, i.e. it makes the
- files that you see correspond to the latest (or any other) commit
- locally present.
-
-* You should push often; there is no real reason not to. Remember that
- even if they are pushed, with the setup above, the commits are (1)
- only in ``bitbucket.org/yourname/pypy``, and (2) in the branch you
- named. Yes, they are publicly visible, but don't worry about someone
- walking around the thousands of repos on bitbucket saying "hah, look
- at the bad coding style of that guy". Try to get into the mindset
- that your work is not secret and it's fine that way. We might not
- accept it as is for PyPy, asking you instead to improve some things,
- but we are not going to judge you.
-
-* The final step is to open a pull request, so that we know that you'd
- like to merge that branch back to the original ``pypy/pypy`` repo.
- This can also be done several times if you have interesting
- intermediate states, but if you get there, then we're likely to
- proceed to the next stage, which is...
-
-* Get a regular account for pushing directly to
- ``bitbucket.org/pypy/pypy`` (just ask and you'll get it, basically).
- Once you have it you can rewrite your file ``.hg/hgrc`` to contain
- ``default = ssh://hg@bitbucket.org/pypy/pypy``. Your changes will
- then be pushed directly to the official repo, but (if you follow these
- rules) they are still on a branch, and we can still review the
- branches you want to merge.
-
-* If you get closer to the regular day-to-day development, you'll notice
- that we generally push small changes as one or a few commits directly
- to the branch ``default``. Also, we often collaborate even if we are
- on other branches, which do not really "belong" to anyone. At this
- point you'll need ``hg merge`` and learn how to resolve conflicts that
- sometimes occur when two people try to push different commits in
- parallel on the same branch. But it is likely an issue for later ``:-)``
-
-.. _bitbucket: https://bitbucket.org/
-
-
-Running PyPy's unit tests
--------------------------
-
-PyPy development always was and is still thoroughly test-driven.
-We use the flexible `py.test testing tool`_ which you can `install independently
-<http://pytest.org/latest/getting-started.html#getstarted>`_ and use for other projects.
-
-The PyPy source tree comes with an inlined version of ``py.test``
-which you can invoke by typing::
-
- python pytest.py -h
-
-This is usually equivalent to using an installed version::
-
- py.test -h
-
-If you encounter problems with the installed version
-make sure you have the correct version installed which
-you can find out with the ``--version`` switch.
-
-You will need the `build requirements`_ to run tests successfully, since many of
-them compile little pieces of PyPy and then run the tests inside that minimal
-interpreter
-
-Now on to running some tests. PyPy has many different test directories
-and you can use shell completion to point at directories or files::
-
- py.test pypy/interpreter/test/test_pyframe.py
-
- # or for running tests of a whole subdirectory
- py.test pypy/interpreter/
-
-See `py.test usage and invocations`_ for some more generic info
-on how you can run tests.
-
-Beware trying to run "all" pypy tests by pointing to the root
-directory or even the top level subdirectory ``pypy``. It takes
-hours and uses huge amounts of RAM and is not recommended.
-
-To run CPython regression tests you can point to the ``lib-python``
-directory::
-
- py.test lib-python/2.7/test/test_datetime.py
-
-This will usually take a long time because this will run
-the PyPy Python interpreter on top of CPython. On the plus
-side, it's usually still faster than doing a full translation
-and running the regression test with the translated PyPy Python
-interpreter.
-
-.. _py.test testing tool: http://pytest.org
-.. _py.test usage and invocations: http://pytest.org/latest/usage.html#usage
-.. _`build requirements`: build.html#install-build-time-dependencies
-
-Special Introspection Features of the Untranslated Python Interpreter
----------------------------------------------------------------------
-
-If you are interested in the inner workings of the PyPy Python interpreter,
-there are some features of the untranslated Python interpreter that allow you
-to introspect its internals.
-
-
-Interpreter-level console
-~~~~~~~~~~~~~~~~~~~~~~~~~
-
-To start interpreting Python with PyPy, install a C compiler that is
-supported by distutils and use Python 2.7 or greater to run PyPy::
-
- cd pypy
- python bin/pyinteractive.py
-
-After a few seconds (remember: this is running on top of CPython), you should
-be at the PyPy prompt, which is the same as the Python prompt, but with an
-extra ">".
-
-If you press
-<Ctrl-C> on the console you enter the interpreter-level console, a
-usual CPython console. You can then access internal objects of PyPy
-(e.g. the :ref:`object space <objspace>`) and any variables you have created on the PyPy
-prompt with the prefix ``w_``::
-
- >>>> a = 123
- >>>> <Ctrl-C>
- *** Entering interpreter-level console ***
- >>> w_a
- W_IntObject(123)
-
-The mechanism works in both directions. If you define a variable with the ``w_`` prefix on the interpreter-level, you will see it on the app-level::
-
- >>> w_l = space.newlist([space.wrap(1), space.wrap("abc")])
- >>> <Ctrl-D>
- *** Leaving interpreter-level console ***
-
- KeyboardInterrupt
- >>>> l
- [1, 'abc']
-
-Note that the prompt of the interpreter-level console is only '>>>' since
-it runs on CPython level. If you want to return to PyPy, press <Ctrl-D> (under
-Linux) or <Ctrl-Z>, <Enter> (under Windows).
-
-Also note that not all modules are available by default in this mode (for
-example: ``_continuation`` needed by ``greenlet``) , you may need to use one of
-``--withmod-...`` command line options.
-
-You may be interested in reading more about the distinction between
-:ref:`interpreter-level and app-level <interpreter-level>`.
-
-pyinteractive.py options
-~~~~~~~~~~~~~~~~~~~~~~~~
-
-To list the PyPy interpreter command line options, type::
-
- cd pypy
- python bin/pyinteractive.py --help
-
-pyinteractive.py supports most of the options that CPython supports too (in addition to a
-large amount of options that can be used to customize pyinteractive.py).
-As an example of using PyPy from the command line, you could type::
-
- python pyinteractive.py --withmod-time -c "from test import pystone; pystone.main(10)"
-
-Alternatively, as with regular Python, you can simply give a
-script name on the command line::
-
- python pyinteractive.py --withmod-time ../../lib-python/2.7/test/pystone.py 10
-
-The ``--withmod-xxx`` option enables the built-in module ``xxx``. By
-default almost none of them are, because initializing them takes time.
-If you want anyway to enable all built-in modules, you can use
-``--allworkingmodules``.
-
-See our :doc:`configuration sections <config/index>` for details about what all the commandline
-options do.
-
-
-.. _trace example:
-
-Tracing bytecode and operations on objects
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-You can use a simple tracing mode to monitor the interpretation of
-bytecodes. To enable it, set ``__pytrace__ = 1`` on the interactive
-PyPy console::
-
- >>>> __pytrace__ = 1
- Tracing enabled
- >>>> x = 5
- <module>: LOAD_CONST 0 (5)
- <module>: STORE_NAME 0 (x)
- <module>: LOAD_CONST 1 (None)
- <module>: RETURN_VALUE 0
- >>>> x
- <module>: LOAD_NAME 0 (x)
- <module>: PRINT_EXPR 0
- 5
- <module>: LOAD_CONST 0 (None)
- <module>: RETURN_VALUE 0
- >>>>
-
-
-Demos
------
-
-The `example-interpreter`_ repository contains an example interpreter
-written using the RPython translation toolchain.
-
-.. _example-interpreter: https://bitbucket.org/pypy/example-interpreter
-
-
-Additional Tools for running (and hacking) PyPy
------------------------------------------------
-
-We use some optional tools for developing PyPy. They are not required to run
-the basic tests or to get an interactive PyPy prompt but they help to
-understand and debug PyPy especially for the translation process.
-
-
-graphviz & pygame for flow graph viewing (highly recommended)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-graphviz and pygame are both necessary if you
-want to look at generated flow graphs:
-
- graphviz: http://www.graphviz.org/Download.php
-
- pygame: http://www.pygame.org/download.shtml
-
-
-py.test and the py lib
-~~~~~~~~~~~~~~~~~~~~~~
-
-The `py.test testing tool`_ drives all our testing needs.
-
-We use the `py library`_ for filesystem path manipulations, terminal
-writing, logging and some other support functionality.
-
-You don't necessarily need to install these two libraries because
-we also ship them inlined in the PyPy source tree.
-
-.. _py library: http://pylib.readthedocs.org/
-
-
-Getting involved
-----------------
-
-PyPy employs an open development process. You are invited to join our
-`pypy-dev mailing list`_ or look at the other :ref:`contact
-possibilities <contact>`. Usually we give out commit rights fairly liberally, so if you
-want to do something with PyPy, you can become a committer. We also run frequent
-coding sprints which are separately announced and often happen around Python
-conferences such as EuroPython or PyCon. Upcoming events are usually announced
-on `the blog`_.
-
-.. _the blog: http://morepypy.blogspot.com
-.. _pypy-dev mailing list: http://mail.python.org/mailman/listinfo/pypy-dev
-
-
-.. _start-reading-sources:
-
-Where to start reading the sources
-----------------------------------
-
-PyPy is made from parts that are relatively independent of each other.
-You should start looking at the part that attracts you most (all paths are
-relative to the PyPy top level directory). You may look at our :doc:`directory reference <dir-reference>`
-or start off at one of the following points:
-
-* :source:`pypy/interpreter` contains the bytecode interpreter: bytecode dispatcher
- in :source:`pypy/interpreter/pyopcode.py`, frame and code objects in
- :source:`pypy/interpreter/eval.py` and :source:`pypy/interpreter/pyframe.py`,
- function objects and argument passing in :source:`pypy/interpreter/function.py`
- and :source:`pypy/interpreter/argument.py`, the object space interface
- definition in :source:`pypy/interpreter/baseobjspace.py`, modules in
- :source:`pypy/interpreter/module.py` and :source:`pypy/interpreter/mixedmodule.py`.
- Core types supporting the bytecode interpreter are defined in :source:`pypy/interpreter/typedef.py`.
-
-* :source:`pypy/interpreter/pyparser` contains a recursive descent parser,
- and grammar files that allow it to parse the syntax of various Python
- versions. Once the grammar has been processed, the parser can be
- translated by the above machinery into efficient code.
-
-* :source:`pypy/interpreter/astcompiler` contains the compiler. This
- contains a modified version of the compiler package from CPython
- that fixes some bugs and is translatable.
-
-* :source:`pypy/objspace/std` contains the :ref:`Standard object space <standard-object-space>`. The main file
- is :source:`pypy/objspace/std/objspace.py`. For each type, the file
- ``xxxobject.py`` contains the implementation for objects of type ``xxx``,
- as a first approximation. (Some types have multiple implementations.)
diff --git a/pypy/doc/how-to-contribute.rst b/pypy/doc/how-to-contribute.rst
deleted file mode 100644
--- a/pypy/doc/how-to-contribute.rst
+++ /dev/null
@@ -1,93 +0,0 @@
-How to contribute to PyPy
-=========================
-
-This page describes how to contribute to the PyPy project. The first thing
-to remember is that PyPy project is very different than most projects out there.
-It's also different from a classic compiler project, so academic courses
-about compilers often don't apply or lead in the wrong direction.
-
-
-Don't just hack
----------------
-
-The first and most important rule how not to contribute to PyPy is
-"just hacking". This won't work. There are two major reasons why not
--- build times are large and PyPy has very thick layer separation which
-make it harder to "just hack a feature".
-
-
-Test driven development
------------------------
-
-Instead, we practice a lot of test driven development. This is partly because
-of very high quality requirements for compilers and partly because there is
-simply no other way to get around such complex project, that will keep you sane.
-There are probably people out there who are smart enough not to need it, we're
-not one of those. You may consider familiarizing yourself with `pytest`_,
-since this is a tool we use for tests.
-This leads to the next issue:
-
-.. _pytest: http://pytest.org/
-
-
-Layers
-------
-
-PyPy has layers. Just like Ogres or onions.
-Those layers help us keep the respective parts separated enough
-to be worked on independently and make the complexity manageable. This is,
-again, just a sanity requirement for such a complex project. For example writing
-a new optimization for the JIT usually does **not** involve touching a Python
-interpreter at all or the JIT assembler backend or the garbage collector.
-Instead it requires writing small tests in
-``rpython/jit/metainterp/optimizeopt/test/test_*`` and fixing files there.
-After that, you can just compile PyPy and things should just work.
-
-The short list of layers for further reading. For each of those layers, a good
-entry point is a test subdirectory in respective directories. It usually
-describes (better or worse) the interfaces between the submodules. For the
-``pypy`` subdirectory, most tests are small snippets of python programs that
-check for correctness (calls ``AppTestXxx``) that will call the appropriate
-part of the interpreter. For the ``rpython`` directory, most tests are small
-RPython interpreters that perform certain tasks. To see how they translate
-to low-level graphs, run them with ``--view``. To see small interpreters
-with a JIT compiler, use ``--viewloops`` option.
-
-* **python interpreter** - it's the part implemented in the ``pypy/`` directory.
- It's implemented in RPython, which is a high level static language with
- classes, garbage collection, just-in-time compiler generation and the ability
- to call C. A cool part about it is that it can be run untranslated, so all
- the tests are runnable without translating PyPy.
-
- **interpreter** contains the interpreter core
-
- **objspace** contains implementations of various objects exported to
- the Python layer
-
- **module** directory contains extension modules written in RPython
-
-* **rpython compiler** that resides in ``rpython/annotator`` and
- ``rpython/rtyper`` directories. Consult `Getting Started with RPython`_
- for further reading
-
-* **JIT generator** lives in ``rpython/jit`` directory. optimizations live
- in ``rpython/jit/metainterp/optimizeopt``, the main JIT in
- ``rpython/jit/metainterp`` (runtime part) and
- ``rpython/jit/codewriter`` (translation-time part). Backends live in
- ``rpython/jit/backend``.
-
-* **garbage collection** lives in ``rpython/memory``
-
-The rest of directories serve specific niche goal and are unlikely a good
-entry point.
-
-
-More documentation
-------------------
-
-* `Getting Started Developing With PyPy`_
-
-* `Getting Started with RPython`_
-
-.. _`Getting Started Developing With PyPy`: getting-started-dev.html
-.. _`Getting started with RPython`: http://rpython.readthedocs.org/en/latest/getting-started.html
diff --git a/pypy/doc/index.rst b/pypy/doc/index.rst
--- a/pypy/doc/index.rst
+++ b/pypy/doc/index.rst
@@ -9,7 +9,7 @@
* If you're interested in trying PyPy out, check out the :doc:`installation instructions <install>`.
-* If you want to help develop PyPy, please have a look at :doc:`how to contribute <how-to-contribute>`
+* If you want to help develop PyPy, please have a look at :doc:`contributing <contributing>`
and get in touch (:ref:`contact`)!
All of the documentation and source code is available under the MIT license,
@@ -63,9 +63,7 @@
.. toctree::
:maxdepth: 1
- getting-started-dev
- how-to-contribute
- you-want-to-help
+ contributing
architecture
configuration
project-ideas
diff --git a/pypy/doc/you-want-to-help.rst b/pypy/doc/you-want-to-help.rst
deleted file mode 100644
--- a/pypy/doc/you-want-to-help.rst
+++ /dev/null
@@ -1,81 +0,0 @@
-You want to help with PyPy, now what?
-=====================================
-
-PyPy is a very large project that has a reputation of being hard to dive into.
-Some of this fame is warranted, some of it is purely accidental. There are three
-important lessons that everyone willing to contribute should learn:
-
-* PyPy has layers. There are many pieces of architecture that are very well
- separated from each other. More about this below, but often the manifestation
- of this is that things are at a different layer than you would expect them
- to be. For example if you are looking for the JIT implementation, you will
- not find it in the implementation of the Python programming language.
-
-* Because of the above, we are very serious about Test Driven Development.
- It's not only what we believe in, but also that PyPy's architecture is
- working very well with TDD in mind and not so well without it. Often
- development means progressing in an unrelated corner, one unittest
- at a time; and then flipping a giant switch, bringing it all together.
- (It generally works out of the box. If it doesn't, then we didn't
- write enough unit tests.) It's worth repeating - PyPy's
- approach is great if you do TDD, and not so great otherwise.
-
-* PyPy uses an entirely different set of tools - most of them included
- in the PyPy repository. There is no Makefile, nor autoconf. More below.
-
-
-Architecture
-------------
-
-PyPy has layers. The 100 miles view:
-
-* :ref:`RPython <rpython:language>` is the language in which we write interpreters. Not the entire
- PyPy project is written in RPython, only the parts that are compiled in
- the translation process. The interesting point is that RPython has no parser,
- it's compiled from the live python objects, which makes it possible to do
- all kinds of metaprogramming during import time. In short, Python is a meta
- programming language for RPython.
-
- The RPython standard library is to be found in the ``rlib`` subdirectory.
-
-* The translation toolchain - this is the part that takes care of translating
- RPython to flow graphs and then to C. There is more in the :doc:`architecture <architecture>`
- document written about it.
-
- It lives in the ``rpython`` directory: ``flowspace``, ``annotator``
- and ``rtyper``.
-
-* Python Interpreter and modules
-
- This is in the ``pypy`` directory. ``pypy/interpreter`` is a standard
- interpreter for Python written in RPython. The fact that it is
- RPython is not apparent at first. Built-in modules are written in
- ``pypy/module/*``. Some modules that CPython implements in C are
- simply written in pure Python; they are in the top-level ``lib_pypy``
- directory. The standard library of Python (with a few changes to
- accomodate PyPy) is in ``lib-python``.
-
-* :ref:`Just-in-Time Compiler (JIT) <rpython:jit>`: we have a tracing JIT that traces the
- interpreter written in RPython, rather than the user program that it
- interprets. As a result it applies to any interpreter, i.e. any
- language. But getting it to work correctly is not trivial: it
- requires a small number of precise "hints" and possibly some small
- refactorings of the interpreter. The JIT itself also has several
- almost-independent parts: the tracer itself in ``rpython/jit/metainterp``, the
- optimizer in ``rpython/jit/metainterp/optimizer`` that optimizes a list of
- residual operations, and the backend in ``rpython/jit/backend/<machine-name>``
- that turns it into machine code. Writing a new backend is a
- traditional way to get into the project.
-
-* Garbage Collectors (GC): as you may notice if you are used to CPython's
- C code, there are no ``Py_INCREF/Py_DECREF`` equivalents in RPython code.
- :ref:`rpython:garbage-collection` is inserted
- during translation. Moreover, this is not reference counting; it is a real
- GC written as more RPython code. The best one we have so far is in
- ``rpython/memory/gc/incminimark.py``.
-
-
-Toolset
--------
-
-xxx
More information about the pypy-commit
mailing list