[CentralOH] State of Python Deployment

Tue Mar 13 19:06:11 EDT 2018

What shuff said!
Also, check out Docker for Mac or Docker for Windows (Win10) or Docker
Toolbox (<Win10).
Pin everything, put it all in the repo, bless the git commit SHA (not the
image), and live happily ever after.

On Sun, Mar 11, 2018 at 6:07 PM Patrick Shuff <patrick.shuff at gmail.com>
wrote:

> The way I would approach this is breaking down the problem into its
> individual components and making sure that whatever I choose always allows
> for a deterministic build/deployment process:
>
> 1.  Choosing the right dependency management system to keep development
> environments to stay consistent and easy (e.g. virtualenv, pipenv, etc)
> 2.  Choosing the right container imaging system to make sure your code and
> dependencies to production is _always_ deterministic (docker, rpm/deb, etc)
> 3.  Choosing system for deploying your code to a server (google app
> engine, heroku, kubernetes, local docker instance, etc)
>
> To expand on my one of my workflows:
>
> 1.  When doing local python development I generally use virtualenv (and
> virtualenvwrapper bash functions) to make it very easy for me to quickly
> spin up my development environment on any platform and pull in all the
> python dependencies without relying on any system libraries.  This is
> important for consistency when you're developing to make sure system-wide
> packages that may differ across operating systems (linux, mac, windows,
> etc) don't impact development.   This is especially important to do based
> on your "conflicting deps" comment.
>
> 2.  Once I am done developing I need to "freeze" the external depencies my
> code relies on, package it up, and create an immutable file system (aka
> "image") of all code and dependencies .  For this I simply freeze the state
> of external python packages and use docker to build it:
>
> pip freeze > requirements.txt
> docker build
>
> The docker build step is important to make sure that whenever I deploy my
> code + dependencies it is always deterministic.  More specifically it means
> I should never have to rely on some outside pypi server to have a specific
> version of the dependencies my code relies on.  This means my code should
> deploy out to my servers today, or in 3 years if I need to migrate to a new
> server and the library versions I was pinned to are long gone.   Hopefully
> we all build our code more often than every 3 years but it is important
> that even if we had to we could without any worries.
>
> If a docker-like deployment style isn't what you're going for another
> alternative that I've used is using pex to create a python executable
> with all your dependencies. <https://github.com/pantsbuild/pex>  It would
> make deploying your software with an (rpm|deb|windows pkg mgr) much easier.
>
>
> 3.  There are countless ways to deploy your code and what you choose may
> influence your decision for the container imaging system in #2.  Spinning
> up local docker instances on a laptop or server running your favorite linux
> distro is extremely easy these days so it is a safe place to start.  If
> your python webapp is something on your local network or your blog docker
> will be fine.  If you need this to scale to many thousands of users and you
> want to using a cloud deployment system google app engine is a decent
> starting point.
>
>
> Sorry that my answers are not very windows focused as I don't have any
> relevant experience there.   Hope this helps!
>
> Good luck! :)
>
> --
> shuff
>
>
>
> On Sun, Mar 11, 2018 at 10:26 AM, Erik Welch <erik.n.welch at gmail.com>
> wrote:
>
>> conda constructor is used to build installers such as the Anaconda
>> distribution, so this probably isn't what you're looking for.
>>
>>
>>>
>>> *(mini)conda + conda env*·         handles binary deps as well
>>> *·         how to pull package deps from an intranet store?*
>>
>>
>> I believe what you're looking for is
>> https://conda.io/docs/user-guide/tasks/create-custom-channels.html .  It
>> is definitely possible to host conda packages internally and to do so on
>> your own.  Some organizations do this without the help of Anaconda (the
>> company I work for).  If you want an enterprise-ready solution and support,
>> then your company can pay my company Anaconda (businesses cater to other
>> businesses, and this is a pain point we have solved for many
>> organizations).  If you want to go it alone, though, the requisite tools
>> and technologies are freely available.
>>
>> I'm sure I'm biased, but it sounds to me like your use case is exactly
>> why conda was created.  There has been lots of good advice on this thread,
>> though, and Docker and pipenv are also good at what they do.
>>
>> Good luck!
>> Erik
>>
>> On Sun, Mar 11, 2018 at 11:37 AM, Eric Floehr <eric at intellovations.com>
>> wrote:
>>
>>> Jason,
>>>
>>> Docker seems like a great way to create completely reproducible
>>> environments. What is your workflow, and are there any tools that make
>>> managing Python environments "pipenv" (or mkvirtualenv/workon) easy?
>>>
>>> -Eric
>>>
>>>
>>> On Fri, Mar 9, 2018 at 12:42 PM, Jason Green <gjigsaw at gmail.com> wrote:
>>>
>>>> I haven't touched virtual environments since I discovered docker in
>>>> 2014.
>>>> If stuck on Windows, check out Docker for Windows. If unable to get to
>>>> Win10, check out Docker ToolBox or Vagrant.
>>>>
>>>>
>>>> On Fri, Mar 9, 2018 at 8:52 AM, Zak Kohler <y2k at y2kbugger.com> wrote:
>>>>
>>>>> I had forgot about this option:
>>>>>
>>>>> https://github.com/conda/constructor
>>>>>
>>>>> Does anyone have experience with conda constructor?
>>>>> _______________________________________________
>>>>> CentralOH mailing list
>>>>> CentralOH at python.org
>>>>> https://mail.python.org/mailman/listinfo/centraloh
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> CentralOH mailing list
>>>> CentralOH at python.org
>>>> https://mail.python.org/mailman/listinfo/centraloh
>>>>
>>>>
>>>
>>> _______________________________________________
>>> CentralOH mailing list
>>> CentralOH at python.org
>>> https://mail.python.org/mailman/listinfo/centraloh
>>>
>>>
>>
>> _______________________________________________
>> CentralOH mailing list
>> CentralOH at python.org
>> https://mail.python.org/mailman/listinfo/centraloh
>>
>>
> _______________________________________________
> CentralOH mailing list
> CentralOH at python.org
> https://mail.python.org/mailman/listinfo/centraloh
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/centraloh/attachments/20180313/07de3bfc/attachment.html>