python package management confusion

[dcs3spp]

On Saturday, 19 January 2019 11:17:19 UTC, dcs3spp  wrote:
> On Saturday, 19 January 2019 07:33:50 UTC, dieter  wrote:
> > dcs3spp via Python-list <python-list at python.org> writes:
> > > On Friday, 18 January 2019 07:39:00 UTC, dieter  wrote:
> > > ...
> > > My situation is similar to the following....
> > >
> > > Assume the following two privately developed projects that I have written, each with their own setup.py:
> > > 1. parent exists in folder $HOME/project/
> > > 2. child exists in folder $HOME/a_different_project/
> > > *child could be also be used by other projects in the future....
> > >
> > > parent has a dependency requirement on child and so child is listed as an install dependency in parent setup.py
> > >
> > > if I try python setup.py develop from parent project it fails to find child dependency on build. 
> > 
> > Do you tell me to have a cyclic dependency structure?
> > 
> > If not, first install "child" in your virtualenv (by whatever
> > means, maybe "python setup.py develop"), then "parent".
> 
> Thanks for your suggestion.No, no cyclic dependency structure. Child does not depend on parent. Child could be used by separate projects in the future. Child will not be cyclic, it will not import projects that import it.
> 
> Installing child first is what I tried yesterday, https://github.com/dcs3spp/setuptools_dependency_example/blob/master/README.md , when trying to understand the suggested approach of using python setup.py develop to pull dependencies.
> 
> The advantage of the devpi approach, adopted earlier, is that this additional complexity is hidden from developers. They do not need to be concerned about the order that they install dependencies, e.g. install child first and then install parent. Developers just run python setup.py develop and it automatically fetches the child dependency. 
> 
> However, the devpi approach requires a private repository to be setup and I am not sure whether it is possible to give access to it for usage in CI cloud technologies, such as gitlab.com. If I stuck with devpi then I would presumably have to follow the route of a bare metal environment for CI server (e.g. Jenkins). Hence, my thoughts on paying subscription to a private pypi cloud repository…..
> 
> 
> My question is, can setuptools be configured to pull in child from a separate git repository when running python setup.py develop from parent folder? I have since found and tried this approach at https://stackoverflow.com/a/53706140/8325270 
> It appears that later versions of setuptools can install via a PEP508 url. I currently trying to investigate this approach…..

After trying PEP508 url approach my conclusions are as follows.

A PEP508 url for a git repository can be used in *install_requires* of *setup.py*. An example is listed below.
```
requires = [
    'parent',
    'kinto-http at git+https://github.com/Kinto/kinto-http.py',
]
...
install_requires=requires
```
The package can then be installed with pip, using ```pip install -e . or pip install .```

However, installation with setuptools is then broken, i.e. ```python setup.py develop``` and ```python setup.py install``` fails. setuptools looks for packages in pypi indexes. To install using setuptools a devpi index would have to be installed and configured or packages would have to installed from a paid for pypi repository in the cloud. Alternatively, developers could manually install each private package dependency individually, prior to running ```python setup.py develop``` for the source package. Unless, there are other alternative(s) such as zc.buildout with mr developer plugin etc.....

If I want to have a Python private project, referencing other private project(s), available under source control and CI via gitlab.com, it seems that I can use the pip approach with PEP508 or use a requirements.txt file containing the git projects referenced as PEP508 urls, i.e. ```pip install -r requirements.txt```.

Confusion, stems from the fact that pip and setuptools dependencies are then not synchronised, i.e. setuptools will break if PEP508 urls are listed for install_requires. Presumably the approach is to use either pip or setuptools but not both?