[AstroPy] importing astropy without reading the config file

[Stephen Bailey]

For the record: setting $XDG_CONFIG_HOME did not work but I don't know
why.  It didn't create any .astropy/ config files in the new location, and
when I moved my $HOME/.astropy out of the way, it still created a new
$HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME
was set to a different location.  I didn't try debugging beyond that.

In the meantime we've moved on with a pragmatic hack that performs a random
sleep between 0 and 60 seconds per rank before doing any astropy imports,
thus spreading out the load.  That buys us a bit more time for development
work; we still plan to use docker/shifter for production work with a
redefined $HOME.

Stephen


On Thu, Oct 26, 2017 at 11:45 PM, Benjamin Alan Weaver <baweaver at lbl.gov>
wrote:

> Hello Stephen,
>
> It looks like XDG_CONFIG_HOME is a possibility, as discussed here:
> http://docs.astropy.org/en/stable/config/index.html, but I'd be concerned
> that other services & applications outside of Python might also see that
> environment variable and look for config files in the directory it
> specifies.  Then again, maybe that's not a bad thing, since the fewer
> things that touch the real $HOME the better.
>
> Kia ora koe,
> Benjamin Alan Weaver
>
> On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey <stephenbailey at lbl.gov>
> wrote:
>
>> We're using astropy in a massively parallel mpi4py program where 4800
>> ranks wake up simultaneously and try to import astropy.  The code itself is
>> installed to a filesystem optimized for the millions (literally!) of tiny
>> file accesses that this causes.  But we're having trouble with all 4800
>> ranks trying to read the astropy config file under our home directory,
>> which is not optimized for such a load.  This intermittently produces the
>> traceback below when one of them gets a corrupted read.
>>
>> Although one could blame the filesystem for not being able to handle an
>> arbitrarily large load, is there any way to mitigate this, e.g.
>>
>> * some environment variable to tell astropy not to load the config file
>> upon import, plus some way of setting the config by hand
>> * some environment variable to tell astropy to load it from a different
>> location than $HOME
>> * other ideas?
>>
>> For those less familiar with MPI, this is a "feature" of MPI that all
>> ranks need to independently load the modules that they need, and they all
>> start nearly simultaneously.  Unlike other parallel programming paradigms,
>> it isn't an option with MPI itself to have one process wake up and import
>> stuff, and then fork other processes that inherit the same environment.
>>  (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that
>> isn't widely implemented/deployed yet.)
>>
>> Our full scale option is to use docker with astropy installed into the
>> image, but even there we have to create a fake $HOME in the docker image so
>> that astropy will get its config files from there instead of reaching out
>> of the container to get them from the real $HOME directory.  It works, but
>> yuck, because it means that nothing else in the program can access $HOME
>> because we've hidden that from the container just to keep astropy imports
>> from abusing it.
>>
>> Thanks for the help,
>>
>> Stephen
>>
>>
>>
>>   File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py",
>> line 21, in <module>
>>     import astropy.io.fits
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line 287,
>> in <module>
>>     log = _init_log()
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100,
>> in _init_log
>>     log._set_defaults()
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493,
>> in _set_defaults
>>     self.setLevel(conf.log_level)
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>> line 278, in __get__
>>     return self()
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>> line 401, in __call__
>>     sec = get_config(self.module)
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>> line 535, in get_config
>>     cobj = configobj.ConfigObj(cfgfn, interpolation=False)
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
>> line 1231, in __init__
>>     self._load(infile, configspec)
>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
>> line 1320, in _load
>>     raise error
>> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00
>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at python.org
>> https://mail.python.org/mailman/listinfo/astropy
>>
>>
>
>
> --
> With great responsibility seldom comes great power.
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org
> https://mail.python.org/mailman/listinfo/astropy
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20171031/71f69395/attachment.html>