Building Python on Cray T3E

[Michael Hudson]

"Mark Hadfield" <m.hadfield at niwa.cri.nz> writes:

[...]
> Oops. The test_sre failure occurred because it couldn't load the
> unicodedata module. I thought I had tried and failed to build
> unicodedata, but in fact I hadn't tried. I did try, it builds OK, and
> test_re succeeds. So cancel my suggestion above. DO patch & build
> _sre.c, DON'T modify Lib/re.py. DO make sure you build unicodedata.c.

Excellent!

[...]
> >>     test test_re crashed -- exceptions.TypeError: 'NoneType' object
> >>     is not callable
> > 
> > This is just weird.  Can you dig a little?
> 
> OK, just a little so far. It's not Cray-related. It occurs on other
> platforms if you modify re.py to select the pre engine. I've seen the
> same exception elsewhere when a function argument is missing. So I'm
> guessing that the problem is that pre does not present exactly the
> same interface as sre.
> 
> It's a moot point for me now that I have sre working, but is the pre
> module still supported? If so should I enter this failure as a bug in
> pre?

I'd guess it's more likely to be a buglet in the test, but I don't
really care :)

You could enter a bug in sf and see if anyone picks it up.  Don't hold
your breath though.

[...]
> > You can get it to not do this by just executing "make python" rather
> > than "make all" or "make".  --disable-shared is a red herring in
> > this context, I think (even in 2.3 where it exists).
> 
> Executing "make python" does indeed bypass attempts to build extension
> modules for dynamic linking, but "make install" and "make test"
> don't. 

Ah yes.

> My ad-hoc solution is to change the following in setup.py
> (class PyBuildExt(build_ext), function build_extensions(self)):
> 
>         # Remove modules that are present on the disabled list
>         self.extensions = [ext for ext in self.extensions
>                            if ext.name not in disabled_module_list]
> 
> to this:
> 
>         self.extensions = []

Probably fair enough.  There should be a better mechanism for this.

I would have thought it would be possible to get the distutils
amchinery to build a static python, i.e. you build a minimal Python
that just contains the libraries needed for distutils (re, os,
os.path, whatever) and then it manages the building of the "real"
python.  I think people doping cross compiles need this sort of
functionality too...

[...]
> Not specifically that one. I think the problem is that running all the
> tests in sequence without reloading the Python interpreter demands a
> lot of memory. On our Compaq Unix machine it takes 67 MB and on the
> Cray it takes 128 MB. This doesn't sound like an excessive amount but
> Cray T3E's are not very smart about swapping and we seem to have some
> sort of load-balancing problem that causes the single-CPU jobs to
> flock to one processor.

Hmm.  Not our problem, then?

> >> My immediate reason for installing Python on the Cray is that I
> >> want to check out SCons, but it *must* work on the Cray to be of
> >> any use to me. At the moment SCons won't work because it requires
> >> md5, which couldn't be compiled because it requires a UINT4 data
> >> type. So I guess I'll either have to get md5 working or persuade
> >> SCons to work without md5 checksums. Oh well...
> > 
> > Yeah, the md5 code that comes with Python is fairly 32 bit specific.
> > However, if you can find some md5-for-cray code in the wild, it's
> > probably not too much effort to hook it up to Python.
> 
> I'll look into that. Alternatively it should be possible to persuade
> SCons to use its TimeStamp module instead of its MD5 module for
> file-change detection. But I haven't quite managed to get that going.
> 
> Still, it's all great fun!

Glad you enjoying it!

Do you feel like drafting a new section for the README?  I think the
Cray notes in there are a bit old.  You might want to liase with
Konrad Hinsen about it, too.

Cheers,
M.

-- 
 As it seems to me, in Perl you have to be an expert to correctly make
 a nested data structure like, say, a list of hashes of instances.  In
 Python, you have to be an idiot not  to be able to do it, because you
 just write it down.             -- Peter Norvig, comp.lang.functional