[IPython-dev] new doc for parallel sessions on clusters

[Johann Cohen-Tanugi]

ok found the problem : I still had the ipython-newapp source code in the 
same dir as the new ipython, I did everything from this new ipython dir,
but at execution it was looking into ipython-newapp. Obviously there is 
something I was not expecting.... Anyway now it works.

sorry again,
Johann

On 06/21/2011 06:41 PM, MinRK wrote:
> On Tue, Jun 21, 2011 at 09:31, Johann Cohen-Tanugi
> <johann.cohentanugi at gmail.com>  wrote:
>> hmmm ok, after the merge is this the right way to get the code :
>>   git clone git://github.com/ipython/ipython.git ipython
>> I removed the whole source directory, and the local install stuff related to
>> python, and I started from scratch using the git command above.
>> then :
>> python setup.py install
>> --prefix=/afs/slac/g/glast/users/cohen/IPYDEV/local/which python
>>
>> perhaps the problem is that there is an ipython installed with the native
>> python that I use above, and I am wrong to assume that having
>> /afs/slac/g/glast/users/cohen/IPYDEV/local/.../site-packages at the top of
>> PYTHONPATH is enough to ensure that it overrides the one installed with
>> python?
> If it was installed with setuptools/easy_install, yes - they inject
> packages at the front sys.path at runtime, ending up ahead of
> PYTHONPATH. easy_installed packages can even come before '.'.
>
> You can check which one you are importing by looking at `IPython.__file__`.
>
> It's possible that I've messed something up, but the source looks
> right (https://github.com/ipython/ipython/blob/master/IPython/core/profileapp.py#L127)
>
>> Anyway, I will continue investigating....
>> sorry for the noise,
>> JOhann
>>
>> On 06/21/2011 05:27 PM, MinRK wrote:
>>> On Tue, Jun 21, 2011 at 07:27, Johann Cohen-Tanugi
>>> <johann.cohen-tanugi at lupm.univ-montp2.fr>    wrote:
>>>> hi there, I tried to build the sphinx doc on my machine after a pull of
>>>> the
>>>> head (after Min announced the merge of newapp into master), and I see the
>>>> following in docs/source/parallel/parallel_process.txt :
>>>> 1/ A trivial typo at :
>>>> :command:`ipcluster` has a notion of Launchers that can start controllers
>>>> and engines with various remote execution schemes.  Currently supported
>>>> models include :command:`ssh`, :command`mpiexec`, PBS-style (Torque,
>>>> SGE),
>>>> and Windows HPC Server.
>>>>
>>>>   |_____>
>>>> should be :command:`mpiexec`
>>>> 2/ I can read the following lines :
>>>>     $ ipython profile create --parallel profile=mpi
>>>> or
>>>>     $ ipython profile create --parallel profile=pbs
>>>>
>>>> while ipython profile create does *not* accept a --parallel option
>>>> AFAICT,
>>>> but rather --cluster or --no-cluster.
>>> Then you do not have current master, because the flag is indeed
>>> '--parallel'.
>>>
>>>> best,
>>>> Johann
>>>>
>>>> On 06/21/2011 07:37 AM, Min RK wrote:
>>>>> Yes, it is going to be standard for the cluster profile and interactive
>>>>> IPython to be different - the profile for the Client is principally for
>>>>> connection info, and the shell profile is for configuring your
>>>>> interactive
>>>>> environment.  There's no reason to change your interactive config just
>>>>> to
>>>>> connect to a different cluster.
>>>>>
>>>>> That said, the default profile of the Client should probably be that of
>>>>> the current application, not just 'default'.
>>>>>
>>>>> -MinRK
>>>>>
>>>>> On Jun 20, 2011, at 22:11, Johann
>>>>> Cohen-Tanugi<johann.cohen-tanugi at lupm.univ-montp2.fr>      wrote:
>>>>>
>>>>>> hi- I don't think we should print the profile name in the default case,
>>>>>>> it's just noise.  I realize we now have a more consistent structure
>>>>>>> for profiles and even the default case is now a profile, but we should
>>>>>>> keep the amount of printed stuff to a minimum in the default cases.
>>>>>>>
>>>>>> Actually I have a question here : I was trying newapp, following Min's
>>>>>> advice, to try to add the LSF support in parallel.apps. From what I
>>>>>> could
>>>>>> gather
>>>>>> I did
>>>>>> ipcluster start -p lsf -n 2
>>>>>> which created profile_lsf in my $HOME/.ipython directory, but then when
>>>>>> I
>>>>>> started another terminal window for the ipython session, I typed
>>>>>> ipython profile=lsf
>>>>>> and this loaded the default profile, so that I had to type :
>>>>>>
>>>>>> from IPython.parallel import Client
>>>>>> c = Client(profile='lsf')
>>>>>>
>>>>>> so that unless this is a bug or an operator mistake, there seems to be
>>>>>> 2
>>>>>> 'profiles' in such a use case : the ipython global one, and the
>>>>>> parallel lsf
>>>>>> one. I find that a bit confusing, and maybe there is a way to merge the
>>>>>> 2?
>>>>>>
>>>>>> best,
>>>>>> Johann