[pypy-dev] Translating PyPy on Windows

[Johan Råde]

Hi everyone!

When I translated PyPy on Windows,
on a laptop with an Intel 3210M processor,
I got the following timings:

   annotate                       ---  464.5 s
   rtype_lltype                   ---  882.7 s
   pyjitpl_lltype                 ---  825.9 s
   backendopt_lltype              ---  259.2 s
   stackcheckinsertion_lltype     ---  309.0 s
   database_c                     ---  349.6 s
   source_c                       ---  502.4 s
   compile_c                      --- 3830.5 s
   ===========================================
   Total:                         --- 7423.8 s

More than half the time is spent compiling C code.
The makefile uses neither precompiled headers nor parallel compilation,
so there is lots of room for improvement.
I tried these two optimizations,
and here are the timings for the C code compilation:

   baseline:   3830.5 s (as above)
   pc:         1301.2 s
   pch:         404.8 s
   pch + pc:    165.8 s

   pc = parallel compilation
   pch = precompiled headers

Using both optimizations you get a factor 23 speed up!

Adding parallel compilation is easy.

Adding precompiled headers is a bit more work;
here I did a quick fix that is not suitable for production code,
but good enough for profiling.

I plan to add parallel compilation on Windows,
along with some other Windows related fixes, to the PyPy code.
I will not add precompiled headers now.

Armin, Maciej: Does that sound OK?

----------------------------------------------------------

The main gain when using precompiled headers comes from the header 
forwarddecl.h.
This header is 17.8 MB and is included in 635 source files.
The amount of C code that is compiled is about 424 MB.
But because of forwarddecl.h, you are in fact compiling another 635 x 
17.8 MB = 11 GB.
Compiling 424 MB or 11 GB of C code - that does make a difference.

But you can not just create a precompiled header that includes 
forwarddecl.h and apply it to all source files.
You should only apply the precompiled header to files that include 
forwarddecl.h to begin with.
These seem to be precisely the source files with names that match one of 
the patterns
data_*.c, nonfuncnodes*.c, implement*.c,  pypy_*.c and rpython_*.c.
But I don't see how to write a nmake rule that only applies to such files.
(nmake does not have as good pattern matching support as GNU make.)

--Johan