[AstroPy] importing astropy without reading the config file

[Stephen Bailey]

We're using astropy in a massively parallel mpi4py program where 4800 ranks
wake up simultaneously and try to import astropy.  The code itself is
installed to a filesystem optimized for the millions (literally!) of tiny
file accesses that this causes.  But we're having trouble with all 4800
ranks trying to read the astropy config file under our home directory,
which is not optimized for such a load.  This intermittently produces the
traceback below when one of them gets a corrupted read.

Although one could blame the filesystem for not being able to handle an
arbitrarily large load, is there any way to mitigate this, e.g.

* some environment variable to tell astropy not to load the config file
upon import, plus some way of setting the config by hand
* some environment variable to tell astropy to load it from a different
location than $HOME
* other ideas?

For those less familiar with MPI, this is a "feature" of MPI that all ranks
need to independently load the modules that they need, and they all start
nearly simultaneously.  Unlike other parallel programming paradigms, it
isn't an option with MPI itself to have one process wake up and import
stuff, and then fork other processes that inherit the same environment.
 (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that
isn't widely implemented/deployed yet.)

Our full scale option is to use docker with astropy installed into the
image, but even there we have to create a fake $HOME in the docker image so
that astropy will get its config files from there instead of reaching out
of the container to get them from the real $HOME directory.  It works, but
yuck, because it means that nothing else in the program can access $HOME
because we've hidden that from the container just to keep astropy imports
from abusing it.

Thanks for the help,

Stephen



  File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py",
line 21, in <module>
    import astropy.io.fits
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line 287, in
<module>
    log = _init_log()
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, in
_init_log
    log._set_defaults()
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, in
_set_defaults
    self.setLevel(conf.log_level)
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
line 278, in __get__
    return self()
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
line 401, in __call__
    sec = get_config(self.module)
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
line 535, in get_config
    cobj = configobj.ConfigObj(cfgfn, interpolation=False)
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
line 1231, in __init__
    self._load(infile, configspec)
  File "/global/common/software/desi/edison/desiconda/20170920-1.2.
0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
line 1320, in _load
    raise error
astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20171026/c922b517/attachment.html>