[AstroPy] importing astropy without reading the config file

[Benjamin Alan Weaver]

Hello Stephen,

It looks like XDG_CONFIG_HOME is a possibility, as discussed here:
http://docs.astropy.org/en/stable/config/index.html, but I'd be concerned
that other services & applications outside of Python might also see that
environment variable and look for config files in the directory it
specifies.  Then again, maybe that's not a bad thing, since the fewer
things that touch the real $HOME the better.

Kia ora koe,
Benjamin Alan Weaver

On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey <stephenbailey at lbl.gov>
wrote:

> We're using astropy in a massively parallel mpi4py program where 4800
> ranks wake up simultaneously and try to import astropy.  The code itself is
> installed to a filesystem optimized for the millions (literally!) of tiny
> file accesses that this causes.  But we're having trouble with all 4800
> ranks trying to read the astropy config file under our home directory,
> which is not optimized for such a load.  This intermittently produces the
> traceback below when one of them gets a corrupted read.
>
> Although one could blame the filesystem for not being able to handle an
> arbitrarily large load, is there any way to mitigate this, e.g.
>
> * some environment variable to tell astropy not to load the config file
> upon import, plus some way of setting the config by hand
> * some environment variable to tell astropy to load it from a different
> location than $HOME
> * other ideas?
>
> For those less familiar with MPI, this is a "feature" of MPI that all
> ranks need to independently load the modules that they need, and they all
> start nearly simultaneously.  Unlike other parallel programming paradigms,
> it isn't an option with MPI itself to have one process wake up and import
> stuff, and then fork other processes that inherit the same environment.
>  (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that
> isn't widely implemented/deployed yet.)
>
> Our full scale option is to use docker with astropy installed into the
> image, but even there we have to create a fake $HOME in the docker image so
> that astropy will get its config files from there instead of reaching out
> of the container to get them from the real $HOME directory.  It works, but
> yuck, because it means that nothing else in the program can access $HOME
> because we've hidden that from the container just to keep astropy imports
> from abusing it.
>
> Thanks for the help,
>
> Stephen
>
>
>
>   File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py",
> line 21, in <module>
>     import astropy.io.fits
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line 287,
> in <module>
>     log = _init_log()
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100, in
> _init_log
>     log._set_defaults()
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493, in
> _set_defaults
>     self.setLevel(conf.log_level)
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
> line 278, in __get__
>     return self()
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
> line 401, in __call__
>     sec = get_config(self.module)
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
> line 535, in get_config
>     cobj = configobj.ConfigObj(cfgfn, interpolation=False)
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
> line 1231, in __init__
>     self._load(infile, configspec)
>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
> line 1320, in _load
>     raise error
> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00
>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org
> https://mail.python.org/mailman/listinfo/astropy
>
>


-- 
With great responsibility seldom comes great power.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20171026/b9899537/attachment-0001.html>