[pypy-svn] rev 1306 - pypy/trunk/doc/funding

[tismer at codespeak.net]

Author: tismer
Date: Mon Sep 15 17:47:44 2003
New Revision: 1306

Added:
   pypy/trunk/doc/funding/B6.1_plan_introduction.txt
Log:
Added an atte,pt of an introduction.
This needs to be chngd quite a lot.
At the moment, it contains all the working packages.
Armin, I need your input, please see
'Low-level targets, tools and releases'

Added: pypy/trunk/doc/funding/B6.1_plan_introduction.txt
==============================================================================
--- (empty file)
+++ pypy/trunk/doc/funding/B6.1_plan_introduction.txt	Mon Sep 15 17:47:44 2003
@@ -0,0 +1,358 @@
+B 6.1 Workplan Introduction
+===============================
+
+The PyPy project has a number of objectives, which
+can be categorized into two major groups:
+
+- development of PyPy itself, as a Python written in Python.
+  This includes code generation for various targets, re-implementation
+  of all builtin Python objects and some extension modules, 
+  and the development of several object spaces, testing and documentation.
+
+- application level code which shows the results of PyPy as a usability
+  study. Generating down-sized code for embedded systems, load balancing
+  in a distributed network, code generators optimized for number crunching
+  on some processor architectures and re-writing numerical Python
+  packages should be able to prove that PyPy is more than an
+  academical exercise but very apropriate for industrial strength
+  applications.
+
+
+*XXX: AT the moment, I'm listing the working packages here.
+I think this is no good. The WPs are now mved out of here.
+The text here gets shortened, the text outside gets
+extended.
+
+
+Working Packages
+===================
+
+The PyPy Interpreter
+---------------------------
+
+The goal is to make a complete Python interpreter that runs under any
+existing Python implementation.
+
+[[WP1]]: Development and Completion of the PyPy Interpreter
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The PyPy interpreter itself should be developed and completed
+as a regular Python program. This package includes all
+parts of CPython tht we don't want to move to [[WP2]].
+Further investigate the unorthodox multimethod
+concepts that the standard object space is based on, and how to hook in
+the bytecode compiler.
+
+[[WP2]]: Translation of CPython into regular Python
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Translate all other parts of CPython which we don't want to
+implement in [[WP1]] into regular Python libraries.
+These ones should also work without PyPy, being just plain-Python
+replacements for existing CPython functionality.
+This includes the bytecode compiler, which definately should
+become a regular Python program, instead of being built
+into the interpreter.
+.
+
+The PyPy Compiler
+---------------------------
+
+
+[[WP3]]: Translation of RPython
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+RPython is a restricted version of Python, which is much
+more statically defined than standard Python. It allows
+to write code that is rather easily translated into something
+else, like C source, Pyrex code, or even passed to
+an assembly code generator.
+The Definition of RPython is slightly in flux and might
+be adjusted during [[WP3]].
+
+The goal is to be able to translate arbitrary RPython source code (e.g. 
+the one produced in [[WP1]] into low-level code (C, Pyrex, Java, others). 
+This includes making a stand-alone, not-PyPy-related tool for general
+optimization of arbitrary but suitably restricted Python application or
+parts thereof.
+
+**Part a): Code Analysis**
+
+Analyse code to produce the relevant typing information. Investigate
+if we can use the annotation object space only or if additional
+AST-based control flow analysis is needed. Give a formal definition
+of RPython.
+
+**Part b): Code Production**
+
+Produce low-level code out of the data gathered in Part a). Again
+investigate how this is best done (AST-guided translation or
+reverse-engeneering of the low-level control flow gathered by the
+annotation object space). Compare different low-level environments that
+we could target (C, Pyrex, others?).
+
+
+[[WP4]]: _`Bootstrapping` PyPy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The goal is to put interpreter ([[WP1]], [[WP2]]) and translator
+([[WP3]]) together.
+
+The interpreter is written as an RPython program. The translator
+has to translate this program into some low-level language.
+The resulting program then needs to be supported by a special
+runtime library. The work-flow of [[WP9]] is repetitive,
+since it will not be possible to "get it right" in the
+first attempt. Analysis and redesign will have to be
+repeated until the result is satisfactory.
+
+**Part a): Specific Analysis and Redesign**
+
+The global translation of PyPy is going to raise particular
+problems, other than more general RPython programs do.
+Since translation of RPython is the core idea of the
+bootstrap process and the main target of the translator,
+we need to investigate and isolate the
+particular problems, and redesign specific parts of PyPy
+to support translation, code generation and optimzation
+better. This will also include a re-iteration of the interface
+design between application level and interpreter level
+until we reach overall convergence.
+
+**Part b): Low Level PyPy Runtime**
+
+In order to give a working environment to the translated RPython
+program, we need to build the low-level-specific **run-time**
+components of PyPy.
+Most notably are the object layout, the memory management,
+and possibly threading support, as well as an
+efficient multimethod dispatch.
+The target language is not yet decided on. We might go different
+paths at the same time.
+If producing C code is a target, important
+parts can be directly re-used from CPython.
+
+
+_`High-performance` PyPy-Python
+-----------------------------------
+The goal is to optimize `Bootstrapping`_ in possibly various ways,
+building on its flexibility to go beyond CPython.
+The main lack of flexibility in CPython stems from the
+fact that all structures are hard-coded in C, and there is
+no abstraction layer. PyPy does provide this abstraction layer,
+since it's RPython implementation is not meant to be
+executed directly, but goes though a code generator which
+produces the actual machine code. This layer is highly configurable.
+
+The associated working packages
+should be done in parallel,since the results are rather dependant
+from each other.
+
+[[WP5]]: Several Object Implementations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+develop **several object implementations** for the same types, as
+explicitly allowed by the standard object space, and develop heuristics
+to switch between implementations during execution. (see `Annex SOI`_)
+Study the efficiency of different approaches.
+
+[[WP6]]: Translator Optimisations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+It should be identified, which **optimizations** would benefit
+from support from the translator. These are the optimizations
+not easily available to CPython because they would require
+large-scale code rewrites. (see `Annex OPT`)
+This includes design-cosiderations including the decision whether
+to use reference counting together with garbage collection,
+or to go for garbage collection, only.
+
+*XXX use a different WP for the next?*
+
+For each issue, work on **several solutions** when no one is obviously
+better than the other ones. The meta-programming underlying [[WP10]]
+-- namely the work on the translator instead of on the resulting code -- is
+what gives us the possibility of actually implementing several very
+different schemes. The outcome of this effort is of course not unique,
+but depends on the specific target of the optimization.
+There will be at least two efforts at the same time:
+
+- optimization towards maximum performance fo an integrated PyPy system
+
+- optimization towards minimal code size for embedded systems.
+
+*add a lot more about this*
+
+[[WP7]]: Integration of Existing Technology
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*XXX should this be split into two WPs?*
+
+There are existing projects, notably Psyco and Stackless,
+which have been traditionally depended on closely
+following CPython's code base.
+
+Psyco should be re-implemented in pure Python, and maybe
+partially in RPython. The intent is to be able to create
+specialized code at run-time. All prior knowledge of the Psyco
+project will be integrated into PyPy.
+
+Stackless Python has implemented high speed multitasking in
+a single thread for C Python in two different ways:
+Continuation passing and stack switching. Both ways of
+task switching can be integrated into PyPy. Furthermore,
+pickling of running programs has been implemented in
+Stackless Python and should enable PyPy for load-balancing
+between different machines.
+
+Rewrite each one as a meta-component that hooks into the
+translator plus a dedicated run-time component. Further
+develop these technologies based on the results gathered in [[WP11]],
+e.g. identify when these technologies would guide specific
+choices among the solutions developed in [[WP10]] and studying
+several solutions in [[WP11]].
+
+
+*XXX The following paragraph is not clear to me,
+how to group it into packages. Armin?*
+
+Low-level targets, tools and releases
+--------------------------------------------
+
+The goal is to identify, among those low-level targets that are in
+widespread use (e.g. workstation usage vs. web server vs.
+high-performance computing vs. memory-starved hand-held device; C/Unix
+vs. Java vs. .NET environment), which ones would benefit most from a
+high-performance Python interpreter. For each of these, focus will be
+given to:
+
+a) develop the translation process, run-time and those optimizations
+   that depend on low-level details.
+
+b) design interfaces for extension modules. Some can be very general
+   (e.g. a pure Python one that should allow generic third-party code to
+   hook into the PyPy interpreter source code without worrying about the
+   translation process). Others depend on the low-level environment and on
+   the choices made for the issues of `High-performance`.
+
+c) (_`APP`)combine different solutions for the different issues discussed in
+   `High-performance`_. Gather statistics with real-work Python applications. Compare the
+   results. This is where the flexibility of the whole project is vital.
+   Typically, very different trade-offs need to be made on different
+   environments.
+
+d) most importantly, develop tools to easily allow third-parties to
+   repeat (APP_) in their own domain and build their own tailored versions of
+   PyPy.
+
+e) release a few official versions pre-tailored for various common
+   environments. Develop in particular a version whose goal is to simulate
+   the existing CPython interpreter to support legacy extension modules. 
+   Investigate if the PyPy core can make internal choices that are very
+   different from CPython's without sacrifying legacy extension modules
+   compatibility.
+
+
+Application Level Targets
+-----------------------------
+
+[[WPxxx]] Load Balancing in a Multi-Processor environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*XXX to be finished, collecting snippets*
+
+[[WPxxx]] Specialized Code for Embedded Devices
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*XXX to be finished, collecting snippets*
+
+[[WPxxx]] Numerical Applications
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+PyPy is extremely flexible, since it supports static
+and dynamic code generation by pluggable processor
+architecture modules. PyPy can use special hardware
+by itself, or it can cross-compile code for target
+machines which will run only a downsized PyPy
+instantiation which is runtime only.
+
+One advantage of using PyPy instead of other compiled
+libraries is its ability to not only support the
+special hardware, but also to adjust to the given
+machine properties, like cache sizes, number of
+parallel FPUs, memory access speed and memory size.
+PyPy is able to probe these parameters and to choose
+an optimum implementation for the particular hardware
+configuration during startup time.
+
+Write specialized code generators which
+support SIMD instruction sets, like SSE2 and AltiVec.
+
+Rewrite a numerical Python package like NumPy
+using RPython. Identify the numerical operations
+in the package which are candidates for parallelization.
+Write an interface that allows to implement these operations
+in either a traditional way or using SIMD instructions,
+dependant from the available capabilities of the current
+code generator.
+The ability of special instructions and other extensions
+should be probed using run-time checks. Psyho should be
+extended to be aware of paralle instructions, and be
+enabled to emit optimized code for that.
+
+
+[[WPxxx]] An Operating System in PyPy
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+*XXX this is what I'm working on at the moment -- the text is
+not yet worked out enough. Another hour...*
+
+
+.. _Annex SOI: 
+
+Annex SOI and others: *where do we put this?*
+
+
+
+Infrastructure
+------------------
+
+*XXX: Numbering these beginning with 100 for the moment
+until we know the exact number of packages*
+
+[[WP101]]: Coordination and Management
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+WP1 continues throughout the duration of the project and is carried out
+by the project coordinator. It involves the collection and monitoring of
+monthly status reports, reporting to the EU, organising meetings
+and maintaining an internal web site. The webiste will make heavy
+use of colaborative tools like Wiki pages, a content management system,
+and issue trackers.
+
+[[WP102]]: Project Documentation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+During the whole project, a set of documents will be maintained
+that documents the current status of the project, results of
+discussions, the planning of new sprints and their status reports,
+as well as the preparation of papers for presentation on
+various congresses.
+
+[[WP103]]: Maintenance of Tools
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+PyPy's own development needs an infrastructure that must continuously
+be kept up-to-date and further developed. This includes maintenance
+of the SubVersion package and tracking of its ongoing development,
+maintenance of the SubVersion respository, extensions like automatic
+documentation extraction, and change notification via mailing lists.
+
+
+[[WP104]]: Quality Control
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The PyPy project has included an exhaustive set of tests for
+all software modules. This has to be extended and maintained
+throughout the whole project. The intent is to be able to
+run all of our internal tests frequently, and to be able to
+run the standard Python test suite, as far as applicable.
+
+[[WP105]]: Synchronisation with Standard Python
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Since Python is being further developed all the time, there is
+a continuous need of keeping PyPy in sync with the future developments
+of CPython, e.g. ways to relate pieces of PyPy source and pieces of CPython
+source. This work is carried out by hand at the moment. We
+do look for existing solutions which allow to automate this
+effort at least partially.