Python 3K or Python 2.9?

[Robin Becker]

Graham Dumpleton wrote:
>.......
> In that blog, Guido says:
> 
> """Concurrency: It seems we're now happily out exploring here. I'm
> looking forward to benchmarks showing that PP or similar (or
> dissimilar!) solutions actually provide a performance gain. Another
> route I'd like to see explored is integrating one such solution into
> an existing web framework (or perhaps as WSGI middleware) so that web
> applications have an easy way out without redesigning their
> architecture."""
.......

Well I have been trying out a system very like PP. For various reasons I prefer 
to write my own scheduler based on more traditional code, but it's very easy to 
get SSH based workers going. Agreed it's a noddy system, but I find the 
following timings for the pp callback demo with start=1, end=200000000.

All the workers are similar to a pp local worker, but some are controlled via 
ssh and hence have to be initialised by an argument like -c"exec input()" which 
downloads some bootstrap code; I find that an easy way to utilize a server which 
already has ssh.

The local workers are clearly dominant, but even though app1,app3,app4 are not 
in the same country they can still be used to provide useful work.

I suspect most people won't want to use  either ssh or socket based workers when 
64 cpu machines arrive. The worker farm topology is also a bit wrong for 
massively parallel stuff.

################################################################
Starting scheduler with 1 workers(1 local)
Partial sum is 0.69314718306 | diff = -2.50006693125e-009
Job execution statistics:
  job count | % of all jobs | job time sum | time per job | worker
        128 |        100.00 |      59.7190 |     0.466555 | local:3676
        128 |        100.00 |      59.7190 |     0.466555 | all local
        128 |        100.00 |      59.7190 |     0.466555 | total
Time elapsed since scheduler creation 59.7820000648
Real average time 0.467046875507


Starting scheduler with 2 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006693125e-009
Job execution statistics:
  job count | % of all jobs | job time sum | time per job | worker
         64 |         50.00 |      31.5300 |     0.492656 | local:324
         64 |         50.00 |      31.2190 |     0.487797 | local:3684
        128 |        100.00 |      62.7490 |     0.490227 | all local
        128 |        100.00 |      62.7490 |     0.490227 | total
Time elapsed since scheduler creation 31.5460000038
Real average time 0.24645312503


Starting scheduler with 3 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006704228e-009
Job execution statistics:
  job count | % of all jobs | job time sum | time per job | worker
         53 |         41.41 |      25.6580 |     0.484113 | local:2392
         52 |         40.63 |      25.5150 |     0.490673 | local:416
         23 |         17.97 |      24.2500 |     1.054348 | ssh:RLUK_robin:20457
        105 |         82.03 |      51.1730 |     0.487362 | all local
        128 |        100.00 |      75.4230 |     0.589242 | total
Time elapsed since scheduler creation 25.7349998951
Real average time 0.20105468668


Starting scheduler with 4 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006704228e-009
Job execution statistics:
  job count | % of all jobs | job time sum | time per job | worker
         46 |         35.94 |      22.2200 |     0.483043 | local:2328
         46 |         35.94 |      22.4670 |     0.488413 | local:3580
         21 |         16.41 |      22.3150 |     1.062619 | ssh:RLUK_robin:20500
         15 |         11.72 |      21.9730 |     1.464867 | ssh:app1:3549
         92 |         71.88 |      44.6870 |     0.485728 | all local
        128 |        100.00 |      88.9750 |     0.695117 | total
Time elapsed since scheduler creation 23.0320000648
Real average time 0.179937500507

Starting scheduler with 5 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006704228e-009
Job execution statistics:
  job count | % of all jobs | job time sum | time per job | worker
         40 |         31.25 |      19.3600 |     0.484000 | local:1376
         41 |         32.03 |      19.7030 |     0.480561 | local:3608
         18 |         14.06 |      19.1090 |     1.061611 | ssh:RLUK_robin:20524
         14 |         10.94 |      19.1110 |     1.365071 | ssh:app1:3596
         15 |         11.72 |      19.0950 |     1.273000 | ssh:app3:44316
         81 |         63.28 |      39.0630 |     0.482259 | all local
        128 |        100.00 |      96.3780 |     0.752953 | total
Time elapsed since server creation 19.9850001335
Real average time 0.156132813543

Starting scheduler with 7 workers(2 local)
Partial sum is 0.69314718306 | diff = -2.50006693125e-009
Job execution statistics:
  job count | % of all jobs | job time sum | time per job | worker
         31 |         24.22 |      14.9540 |     0.482387 | local:2588
         31 |         24.22 |      15.0630 |     0.485903 | local:472
         13 |         10.16 |      14.0790 |     1.083000 | ssh:RLUK_robin:20548
          9 |          7.03 |      14.7680 |     1.640889 | ssh:app1:3626
         11 |          8.59 |      14.3260 |     1.302364 | ssh:app3:44344
         15 |         11.72 |      14.2990 |     0.953267 | ssh:app4:61396
         18 |         14.06 |      14.4050 |     0.800278 | ssh:app5:1258
         62 |         48.44 |      30.0170 |     0.484145 | all local
        128 |        100.00 |     101.8940 |     0.796047 | total
Time elapsed since scheduler creation 15.2030000687
Real average time 0.118773438036
################################################################
-- 
Robin Becker