[AstroPy] importing astropy without reading the config file

Tue Oct 31 15:59:51 EDT 2017

Hello Stephen,

According to the documentation,
http://docs.astropy.org/en/stable/config/index.html, $XDG_CONFIG_HOME/astropy
must *already* exist. You can take a look at
https://github.com/astropy/astropy/blob/master/astropy/config/paths.py to
see the logic, but to summarize:

1. $XDG_CONFIG_HOME must be set.

2. $XDG_CONFIG_HOME/astropy/ must *exist*.  Note that's 'astropy' not
'.astropy'.

3. The configuration files go in $XDG_CONFIG_HOME/astropy/, NOT
$XDG_CONFIG_HOME/astropy/config/.  This one confused me!

You might also want to consider setting XDG_CONFIG_CACHE, so that cache
files are written to some other filesystem than $HOME.

Kia ora koe,
Benjamin Alan Weaver

On Tue, Oct 31, 2017 at 12:01 PM, Stephen Bailey <stephenbailey at lbl.gov>
wrote:

> For the record: setting $XDG_CONFIG_HOME did not work but I don't know
> why.  It didn't create any .astropy/ config files in the new location, and
> when I moved my $HOME/.astropy out of the way, it still created a new
> $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME
> was set to a different location.  I didn't try debugging beyond that.
>
> In the meantime we've moved on with a pragmatic hack that performs a
> random sleep between 0 and 60 seconds per rank before doing any astropy
> imports, thus spreading out the load.  That buys us a bit more time for
> development work; we still plan to use docker/shifter for production work
> with a redefined $HOME.
>
> Stephen
>
>
> On Thu, Oct 26, 2017 at 11:45 PM, Benjamin Alan Weaver <baweaver at lbl.gov>
> wrote:
>
>> Hello Stephen,
>>
>> It looks like XDG_CONFIG_HOME is a possibility, as discussed here:
>> http://docs.astropy.org/en/stable/config/index.html, but I'd be
>> concerned that other services & applications outside of Python might also
>> see that environment variable and look for config files in the directory it
>> specifies.  Then again, maybe that's not a bad thing, since the fewer
>> things that touch the real $HOME the better.
>>
>> Kia ora koe,
>> Benjamin Alan Weaver
>>
>> On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey <stephenbailey at lbl.gov>
>> wrote:
>>
>>> We're using astropy in a massively parallel mpi4py program where 4800
>>> ranks wake up simultaneously and try to import astropy.  The code itself is
>>> installed to a filesystem optimized for the millions (literally!) of tiny
>>> file accesses that this causes.  But we're having trouble with all 4800
>>> ranks trying to read the astropy config file under our home directory,
>>> which is not optimized for such a load.  This intermittently produces the
>>> traceback below when one of them gets a corrupted read.
>>>
>>> Although one could blame the filesystem for not being able to handle an
>>> arbitrarily large load, is there any way to mitigate this, e.g.
>>>
>>> * some environment variable to tell astropy not to load the config file
>>> upon import, plus some way of setting the config by hand
>>> * some environment variable to tell astropy to load it from a different
>>> location than $HOME
>>> * other ideas?
>>>
>>> For those less familiar with MPI, this is a "feature" of MPI that all
>>> ranks need to independently load the modules that they need, and they all
>>> start nearly simultaneously.  Unlike other parallel programming paradigms,
>>> it isn't an option with MPI itself to have one process wake up and import
>>> stuff, and then fork other processes that inherit the same environment.
>>>  (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that
>>> isn't widely implemented/deployed yet.)
>>>
>>> Our full scale option is to use docker with astropy installed into the
>>> image, but even there we have to create a fake $HOME in the docker image so
>>> that astropy will get its config files from there instead of reaching out
>>> of the container to get them from the real $HOME directory.  It works, but
>>> yuck, because it means that nothing else in the program can access $HOME
>>> because we've hidden that from the container just to keep astropy imports
>>> from abusing it.
>>>
>>> Thanks for the help,
>>>
>>> Stephen
>>>
>>>
>>>
>>>   File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py",
>>> line 21, in <module>
>>>     import astropy.io.fits
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line
>>> 287, in <module>
>>>     log = _init_log()
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100,
>>> in _init_log
>>>     log._set_defaults()
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493,
>>> in _set_defaults
>>>     self.setLevel(conf.log_level)
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>>> line 278, in __get__
>>>     return self()
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>>> line 401, in __call__
>>>     sec = get_config(self.module)
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>>> line 535, in get_config
>>>     cobj = configobj.ConfigObj(cfgfn, interpolation=False)
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
>>> line 1231, in __init__
>>>     self._load(infile, configspec)
>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
>>> line 1320, in _load
>>>     raise error
>>> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00
>>>
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at python.org
>>> https://mail.python.org/mailman/listinfo/astropy
>>>
>>>
>>
>>
>> --
>> With great responsibility seldom comes great power.
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at python.org
>> https://mail.python.org/mailman/listinfo/astropy
>>
>>
>
> _______________________________________________
> AstroPy mailing list
> AstroPy at python.org
> https://mail.python.org/mailman/listinfo/astropy
>
>

-- 
With great responsibility seldom comes great power.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20171031/f40a6809/attachment-0001.html>