[AstroPy] importing astropy without reading the config file

Tue Oct 31 16:00:57 EDT 2017

Hello Stephen,

PS, just to be clear, when I followed all three steps, it did work for me.

Kia ora koe,
Benjamin Alan Weaver

On Tue, Oct 31, 2017 at 12:59 PM, Benjamin Alan Weaver <baweaver at lbl.gov>
wrote:

> Hello Stephen,
>
> According to the documentation, http://docs.astropy.org/en/stable/config/
> index.html, $XDG_CONFIG_HOME/astropy must *already* exist. You can take a
> look at https://github.com/astropy/astropy/blob/master/astropy/
> config/paths.py to see the logic, but to summarize:
>
> 1. $XDG_CONFIG_HOME must be set.
>
> 2. $XDG_CONFIG_HOME/astropy/ must *exist*.  Note that's 'astropy' not
> '.astropy'.
>
> 3. The configuration files go in $XDG_CONFIG_HOME/astropy/, NOT
> $XDG_CONFIG_HOME/astropy/config/.  This one confused me!
>
> You might also want to consider setting XDG_CONFIG_CACHE, so that cache
> files are written to some other filesystem than $HOME.
>
> Kia ora koe,
> Benjamin Alan Weaver
>
>
>
> On Tue, Oct 31, 2017 at 12:01 PM, Stephen Bailey <stephenbailey at lbl.gov>
> wrote:
>
>> For the record: setting $XDG_CONFIG_HOME did not work but I don't know
>> why.  It didn't create any .astropy/ config files in the new location, and
>> when I moved my $HOME/.astropy out of the way, it still created a new
>> $HOME/.astropy/config/ directory upon import even though $XDG_CONFIG_HOME
>> was set to a different location.  I didn't try debugging beyond that.
>>
>> In the meantime we've moved on with a pragmatic hack that performs a
>> random sleep between 0 and 60 seconds per rank before doing any astropy
>> imports, thus spreading out the load.  That buys us a bit more time for
>> development work; we still plan to use docker/shifter for production work
>> with a redefined $HOME.
>>
>> Stephen
>>
>>
>> On Thu, Oct 26, 2017 at 11:45 PM, Benjamin Alan Weaver <baweaver at lbl.gov>
>> wrote:
>>
>>> Hello Stephen,
>>>
>>> It looks like XDG_CONFIG_HOME is a possibility, as discussed here:
>>> http://docs.astropy.org/en/stable/config/index.html, but I'd be
>>> concerned that other services & applications outside of Python might also
>>> see that environment variable and look for config files in the directory it
>>> specifies.  Then again, maybe that's not a bad thing, since the fewer
>>> things that touch the real $HOME the better.
>>>
>>> Kia ora koe,
>>> Benjamin Alan Weaver
>>>
>>> On Thu, Oct 26, 2017 at 8:30 PM, Stephen Bailey <stephenbailey at lbl.gov>
>>> wrote:
>>>
>>>> We're using astropy in a massively parallel mpi4py program where 4800
>>>> ranks wake up simultaneously and try to import astropy.  The code itself is
>>>> installed to a filesystem optimized for the millions (literally!) of tiny
>>>> file accesses that this causes.  But we're having trouble with all 4800
>>>> ranks trying to read the astropy config file under our home directory,
>>>> which is not optimized for such a load.  This intermittently produces the
>>>> traceback below when one of them gets a corrupted read.
>>>>
>>>> Although one could blame the filesystem for not being able to handle an
>>>> arbitrarily large load, is there any way to mitigate this, e.g.
>>>>
>>>> * some environment variable to tell astropy not to load the config file
>>>> upon import, plus some way of setting the config by hand
>>>> * some environment variable to tell astropy to load it from a different
>>>> location than $HOME
>>>> * other ideas?
>>>>
>>>> For those less familiar with MPI, this is a "feature" of MPI that all
>>>> ranks need to independently load the modules that they need, and they all
>>>> start nearly simultaneously.  Unlike other parallel programming paradigms,
>>>> it isn't an option with MPI itself to have one process wake up and import
>>>> stuff, and then fork other processes that inherit the same environment.
>>>>  (For those who are MPI experts, I'm aware of MPI_Comm_spawn, but that
>>>> isn't widely implemented/deployed yet.)
>>>>
>>>> Our full scale option is to use docker with astropy installed into the
>>>> image, but even there we have to create a fake $HOME in the docker image so
>>>> that astropy will get its config files from there instead of reaching out
>>>> of the container to get them from the real $HOME directory.  It works, but
>>>> yuck, because it means that nothing else in the program can access $HOME
>>>> because we've hidden that from the container just to keep astropy imports
>>>> from abusing it.
>>>>
>>>> Thanks for the help,
>>>>
>>>> Stephen
>>>>
>>>>
>>>>
>>>>   File "/global/common/software/desi/users/jguy/desispec/py/desispec/io/brick.py",
>>>> line 21, in <module>
>>>>     import astropy.io.fits
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/__init__.py", line
>>>> 287, in <module>
>>>>     log = _init_log()
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 100,
>>>> in _init_log
>>>>     log._set_defaults()
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/logger.py", line 493,
>>>> in _set_defaults
>>>>     self.setLevel(conf.log_level)
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>>>> line 278, in __get__
>>>>     return self()
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>>>> line 401, in __call__
>>>>     sec = get_config(self.module)
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/config/configuration.py",
>>>> line 535, in get_config
>>>>     cobj = configobj.ConfigObj(cfgfn, interpolation=False)
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
>>>> line 1231, in __init__
>>>>     self._load(infile, configspec)
>>>>   File "/global/common/software/desi/edison/desiconda/20170920-1.2.
>>>> 0-spec/conda/lib/python3.6/site-packages/astropy/extern/configobj/configobj.py",
>>>> line 1320, in _load
>>>>     raise error
>>>> astropy.extern.configobj.configobj.ParseError: Invalid line ('\x00\x00
>>>>
>>>>
>>>> _______________________________________________
>>>> AstroPy mailing list
>>>> AstroPy at python.org
>>>> https://mail.python.org/mailman/listinfo/astropy
>>>>
>>>>
>>>
>>>
>>> --
>>> With great responsibility seldom comes great power.
>>>
>>> _______________________________________________
>>> AstroPy mailing list
>>> AstroPy at python.org
>>> https://mail.python.org/mailman/listinfo/astropy
>>>
>>>
>>
>> _______________________________________________
>> AstroPy mailing list
>> AstroPy at python.org
>> https://mail.python.org/mailman/listinfo/astropy
>>
>>
>
>
> --
> With great responsibility seldom comes great power.
>

-- 
With great responsibility seldom comes great power.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/astropy/attachments/20171031/7fe96de5/attachment.html>