Building Python on Cray T3E

[Michael Hudson]

"Mark Hadfield" <m.hadfield at niwa.co.nz> writes:

> "Michael Hudson" <mwh at python.net> wrote in message
> news:lkhelg74ts.fsf at pc150.maths.bris.ac.uk...
> 
> > So does the resulting Python actually work after this?  We got this
> > far with 2.2.1c1, but you never told me if the dang thing worked or
> > not...
> 
> I'm not sure who this question was directed to,

Kalle Svensson.  He tried to build on a Cray (a J90, I think) when
2.2.1c1 came out, but didn't have the time to finish the job off.

> but here are the results of my attempts to build Python 2.2.1 on a
> Cray T3E over the last few days. The summary is: it can be built and
> installed and it runs several simple scripts, but several modules
> are broken to a greater or lesser extent. It's not yet clear to me
> how useful it will be or how much work is required to fix things.

That's cool.

> The procedure is the usual "configure; make; make install" with the
> following variations and notes:
> 
>   - The compiler (found automatically by the configure script) is
>   Cray's cc. We don't have gcc on the NIWA machine and a note from
>   Konrad Hinsen in README warns against using it anyway.

That note is probably pretty old; I don't know if it's still relavent.

>   - As has been discussed before on this group, Python's unicode
>   module cannot be built because the Cray compiler lacks a 16-bit
>   character datatype (something like that--don't trust me on the
>   details here). So we disable it with "configure
>   --enable-unicode=ucs4".

That doesn't actually disable unicode (you want --disable-unicode for
that).  *Lots* of tests fail in --disable-unicode builds, mostly for
very shallow reasons (u'' literals not being supported for the most
part).

>   - As noted earlier in this thread, there is a C-function name clash
>   in the _sre module, which can be easily fixed. However, recall that
>   sre is one of two alternative regular-expression engines for the re
>   module, the other engine being pre. It turns out that sre does not
>   pass its tests so the path of least resistance is is to modify
>   Lib/re.py to use the pre engine (the change is trivial) and comment
>   out the _sre entry in Modules/Setup.

How does it fail?  Is it the "NoneType not callable" failure below.

That's an odd one.

>   - The Cray compiler does not support dynamic linking, so extensions
>   must be explicitly listed in Modules/Setup. Here is the list of
>   extensions I have built & linked successfully:
> 
>     posix posixmodule.c
>     new newmodule.c
>     array arraymodule.c
>     cmath cmathmodule.c
>     math mathmodule.c
>     struct structmodule.c
>     time timemodule.c
>     operator operator.c
>     _weakref _weakref.c
>     _codecs _codecsmodule.c
>     _testcapi _testcapimodule.c
>     _symtable symtablemodule.c
>     strop stropmodule.c
>     regex regexmodule.c regexpr.c
>     pcre pcremodule.c pypcre.c
>     _locale _localemodule.c
>     fcntl fcntlmodule.c
>     pwd pwdmodule.c
>     grp grpmodule.c
>     errno errnomodule.c
>     select selectmodule.c
>     xreadlines xreadlinesmodule.c
>     _socket socketmodule.c
>     crypt cryptmodule.c
>     termios termios.c
>     timing timingmodule.c
>     rotor rotormodule.c
>     syslog syslogmodule.c
>     dbm dbmmodule.c
>     binascii binascii.c
>     parser parsermodule.c
>     cStringIO cStringIO.c
>     cPickle cPickle.c

That's a fairly impressive list.

Did the unicodedata module fail to build, or did you not try it?

>   - With the above modules included, Python is built
>   successfully. There are quite a few warnings, which I haven't
>   checked out yet. I think that some of them come home to roost in the
>   tests.

Quite likely.  Though Python is more-or-less 64 bit clean (runs on
alphas, for isntance).  Having 64-bit chars is probably a surprise to
some of the code, though...

>   - After building Python, make runs setup.py, which attempts (among
>   other things) to build extensions not already built & linked
>   statically. This is a futile but harmless exercise; the inevitable
>   failures do not cause make to abort. I have seen elsewhere (in
>   connection with Cygwin Python) a reference to a "--disable-shared"
>   option to configure that I presume would inhibit this. However this
>   is ignored by configure in Python 2.2.1.

You can get it to not do this by just executing "make python" rather
than "make all" or "make".  --disable-shared is a red herring in this
context, I think (even in 2.3 where it exists).

>   - Testing is a somewhat depressing experience. On the NIWA machine,
>   a simple "make test" command fails to complete after 12 hours and
>   thrashes the swap space, with the side effect of inhibiting all
>   network access for 10 seconds out of every 30. 

!  Is this test_longexp, by any chance?

>   (They don't call these things supercomputers for nothing. I bet
>   you can't do *that* on your $1000 Linux box!) By running "make
>   test" for a while then completing the tests one at a time, I have
>   established that the following tests fail:

Most of the skips are harmless.

>     test test_al skipped -- No module named al
>     test test_audioop skipped -- No module named audioop
>     test test_binhex failed -- CRC error
>     test test_bsddb skipped -- No module named bsddb
>     test test_cd skipped -- No module named cd
>     test test_cl skipped -- No module named cl
>     test test_curses skipped -- No module named _curses
>     test test_descr skipped -- No module named xxsubtype
>     test test_dl skipped -- No module named dl
>     test test_fcntl crashed -- IO Error

Hmm.

>     test test_fpformat failed -- AssertionError: '-0' != '0'

Do Crays still have eccentric fp behaviour?

>     test test_hmac failed -- No module named md5
>     test test_hotshot skipped -- No module named _hotshot
>     test test_imageop skipped -- No module named imageop
>     test test_imgfile skipped -- No module named imgfile
>     test test_linuxaudiodev skipped -- No module named linuxaudiodev
>     test test_locale skipped -- test locale en_US not supported
>     test test_longexp crashed -- exceptions.MemoryError:

Oh, so this isn't the one that thrashes for days...

I could imagine that this test would consume serious memory on a
cray...

>     test test_md5 skipped -- No module named md5
>     test test_minidom skipped -- No module named pyexpat
>     test test_mmap skipped -- No module named mmap
>     test test_nis skipped -- No module named nis
>     test test_ntpath skipped -- No module named nt
>     test test_openpty skipped -- No openpty() available.
>     test test_pep247 skipped -- No module named md5
>     test test_poll skipped -- select.poll not defined
>     test test_pty skipped -- Pseudo-terminals (seemingly) not
>       functional.
>     test test_pyexpat skipped -- No module named pyexpat
>     test test_re crashed -- exceptions.TypeError: 'NoneType' object is
>       not callable

This is just wierd.  Can you dig a little?

>     test test_rgbimg skipped -- No module named rgbimg
>     test test_rotor produced unexpected output:
>     test test_sax skipped -- no XML parsers available
>     test test_sha skipped -- No module named sha
>     test test_socket crashed -- socket.error: unknown address family
>     test test_socket_ssl skipped -- Use of the `network' resource
>       not enabled
>     test test_socketserver skipped -- Use of the `network' resource
>       not enabled
>     test test_sre crashed -- exceptions.UnicodeError: \N escapes not
>       supported (can't load unicodedata module)

This is probably shallow.  You could try hacking the unicode bits of
the test out.

>     test test_sunaudiodev skipped -- No module named sunaudiodev
>     test test_sundry skipped -- No module named md5
>     test test_ucn skipped -- No module named unicodedata
>     test test_unicode crashed -- exceptions.UnicodeError: \N escapes
>       not supported (can't load unicodedata module)
>     test test_unicode_file skipped -- No Unicode filesystem semantics
>       on this platform.
>     test test_unicodedata skipped -- No module named sha
>     test test_urllib2 skipped -- No module named md5
>     test test_winreg skipped -- No module named winreg
>     test test_winsound skipped -- No module named winsound
>     test test_zipfile skipped -- No module named zlib
>     test test_zlib skipped -- No module named zlib
> 
>   Not all of the test failures are of any significance, of course. (I
>   didn't really expect Windows registry operations to work.) But the
>   failure in test_re is a little worrying...
> 
> My immediate reason for installing Python on the Cray is that I want
> to check out SCons, but it *must* work on the Cray to be of any use to
> me. At the moment SCons won't work because it requires md5, which
> couldn't be compiled because it requires a UINT4 data type. So I guess
> I'll either have to get md5 working or persuade SCons to work without
> md5 checksums. Oh well...

Yeah, the md5 code that comes with Python is fairly 32 bit specific.
However, if you can find some md5-for-cray code in the wild, it's
probably not too much effort to hook it up to Python.

Cheers,
M.

-- 
 Very clever implementation techniques are required to implement this
 insanity correctly and usefully, not to mention that code written
 with this feature used and abused east and west is exceptionally
 exciting to debug.       -- Erik Naggum on Algol-style "call-by-name"