[Distutils] Serverside Dependency Resolution and Virtualenv Build Server

[Wes Turner]

* It is [currently [#PEP426JSONLD)] necessary to run setup.py with each
given destination platform, because parameters are expanded within the
scope of setup.py.
  * Because of this, client side dependency resolution (with a given
platform) is currently the only viable option for something like this

...

* Build: Docker, Tox (Dox) to build package(s)
  * Each assembly of packages is / could be a package with a setup.py
(and/or a requirements.txt)
     * And tests:
       * http://conda.pydata.org/docs/building/meta-yaml.html#test-section
* Release: DevPi
   * http://doc.devpi.net/latest/
* conda env environment.yml YAML:
http://conda.pydata.org/docs/using/envs.html
  * [x] conda packages
  * [x] pip packages
  * [ ] system packages (configuration management)

And then, really, Is there a stored version of this instance of a named
Docker image?
#reproducibility #linkedreproducibility

On Sat, Nov 7, 2015 at 8:37 AM, Thomas Güttler <guettliml at thomas-guettler.de
> wrote:

> I wrote down a tought about Serverside Dependency Resolution and
> Virtualenv Build Server
>
>
> What do you think?
>
> Latest version: https://github.com/guettli/virtualenv-build-server
>
> virtualenv-build-server
> #######################
>
> Rough roadmap how a server to build virtualenvs for the python programming
> language could be implemented.
>
> Highlevel goal
> --------------
>
> Make creating new virtual envionments for the Python programming language
> easy and fast.
>
> Input: fuzzy requirements like this: django>=1.8, requests=>2.7
>
> Output: virtualenv with packages installed.
>
> Two APIs
> ------------
>
>  #. Resolve fuzzy requirements to a fixed set of packages with exactly
> pinned versions.
>  #. Read fixed set of packages. Build virtualenv according to given
> platform.
>
>
> Steps
> -----
>
> Steps:
>
> #. Client sends list of fuzzy requirements to server:
>
>    * I need: django>=1.8, requests=>2.7, ...
>
>
> #. Server solves the fuzzy requirements to a fixed set of requirememnts:
> django==1.8.2, requests==2.8.1, ...
>
> #. Client reads the fixed set of requirements.
>
> #. Optional: Client sends fixed set of requirements to the server. Telling
> him the plattform
>
>    * My platform: sys.version==2.7.6 and sys.platform=linux2
>
> #. Server builds a virtualenv according to the fixed set of requirements.
>
> #. Server sends the environment to the client
>
> #. Client unpacks the data and has a usable virtualenv
>
> Benefits
> --------
>
> Speed:
>
> * There is only one round-trip from client to server. If the dependencies
> get resolved on the client the client would need to download the available
> version information.
> * Caching: If the server gets input parameters (fuzzy requirements and
> platform information) which he has seen before, he can return the cached
> result from the previous request.
>
> Possible Implementations
> ------------------------
>
> APIs
> ====
> Both APIs could be implementated by a webservice/Rest interface passing
> json or yaml.
>
> Serverside
> ==========
>
>
> Implementation Strategie "PostgreSQL"
> .....................................
>
> Since the API is de-coupled from the internals the implementation could be
> exchanged without the need for changes at the client side.
>
> I suggest using the PostgreSQL und resolving the dependcy graph using SQL
> (WITH RECURSIVE).
>
> The package and version data gets stored in PostgreSQL via ORM (Django or
> SQLAlchemy).
>
> The version numbers need to be normalized to ascii to allow fast
> comparision.
>
> Related: https://www.python.org/dev/peps/pep-0440/
>
> Implementation Strategie "Node.js"
> ..................................
>
> I like python, but I am not married with it. Why not use a different tools
> that is already working? Maybe the node package manager:
> https://www.npmjs.com/
>
> Questions
> ---------
> Are virtualenv relocatable? AFAIK they are not.
>
> General Thoughts
> ----------------
>
>  * Ignore Updates. Focus on creating new virtualenvs. The server can do
> caching and that's why I prefer creating virtualenvs which never get
> updated. They get created and removed (immutable).
>
>
> I won't implement it
> --------------------
>
> This idea is in the public domain. If you are young and brave or old and
> wise: Go ahead, try to implement it. Please communicate early and often.
> Ask on mailing-lists or me for feedback. Good luck :-)
>
> I love feedback
> ---------------
>
> Please tell me want you like or dislike:
>
>  * typos and spelling stuff (I am not a native speaker)
>  * alternative implementation strategies.
>  * existing software which does this (even if implemented in a different
> programming language).
>  * ...
>
> --
> http://www.thomas-guettler.de/
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20151110/a31d5637/attachment.html>