[Mailman-Developers] [MM3-users] Re: Rolling releases for Container Images and Funding Campaign for Mailman

[Abhilash Raj]

On Wed, Nov 8, 2017, at 04:38 AM, Simon Hanna wrote:
> On 11/08/2017 12:03 AM, Abhilash Raj wrote:
> > New images are available on quay.io and, moving forward, the rest of
> > the image builds will also be moved to Quay[4][5].
> Since docker deployment is currently the recommended install, with 
> manual installs using the master branches for "experts" I wonder why you 
> are using Quay.

For those who don't know about it, Quay is a container registry offering
from CoreOS. The main reason why I chose to move to Quay, from DockerHub
that we are currently using, is security.

Some parts of both Dockerhub and Quay are open source and some aren't.
There isn't a
very clear boundary/transparency in what technologies they use. Both
are free to use for open source projects. However, Quay provides an
improved
security model with RBAC to push images from build server, without
having to put my account password on the build server. It also includes
a
(open source) security scanner for images, which looks for
vulnerabilities in packages
included in the image so that I can re-build them if one is found.

There are open source registries available, most popular of which I know
is provided by Gitlab. However, I feel Quay is better for my current
workflow, which involves lots of bash scripts for building and testing 
images on public CI system for rolling releases.

Given more free cycles, I do intend to have a better build pipeline and
possibly
use Gitlab's own registry to distribute images.

> It looks like it's a non-free system. Why use Quay but refuse to use a 
> non-free translation system like transifex?
> > These images are built using git-heads *only* if they are passing our
> > test suite and are re-generated weekly. You should be aware that while
> > all these components are tested with their individual test suites, their
> > combination might sometimes not be stable. This will get you
> > updates/bug-fixes much faster :)
> This contradicts what you said in my merge request for Postorius
> https://gitlab.com/mailman/postorius/merge_requests/232#note_43388756
> 
> You either use released versions, or make sure that the master branches 
> are always compatible. You can't have both with the current development 
> model.

Not exactly, I still believe all our git-heads should be compatible with
released (i.e. stable) versions of dependencies, regardless of the fact 
that we maintain those dependencies.

Also, there is at least one test per-project which tests with git-heads
of dependencies we maintain, so that we know if we make any incompatible
changes. 

I am not sure what in the current development model prevents us
to do both?

The story for container images is different than testing with git-heads.
We need better integration testing for the entire suite to be stable in
conjunction, rather than independently. In fact, containers might be the
perfect way to actually do the integration testing.

> > If this campaign succeeds, here is a road map of what I intend to get
> > done:
> Sounds great, what is the limit where the campaign succeeds? What will 
> happen to the funds if it doesn't?
> >    - Add Admin Dashboard project from GSoC 2014 (maybe?)
> Didn't find anything using google. What is that project about?

https://www.google-melange.com/archive/gsoc/2015/orgs/gnu_mailman/projects/bhaveshgoyal093.html

> > - Add better testing of container images and provide deployment
> >    instructions for Kubernetes & Docker Swarm
> To be honest, I don't think we should all in on containers.
> Yes they are nice, but I guess most people would prefer distro packages.
> Even if people want containers, I for instance would prefer plain 
> systemd-nspawn

Like Barry mentioned, it doesn't have to be all-or-nothing. There is
work
being done on distro packaging for Debian and I am ready to help others
trying
to do the same for other distros!

Systemd-nspawn isn't really a production grade tool to use containers. I
am
not sure about how much it adheres to the OCI specs, but if it does, it
should
not matter which deployment tool you are using with images.

I wouldn't mind adding configurations for systemd-nspawn in the
documentation,
contributions are most welcome.

> > - Improve the container images to work with new micro-services
> > architecture,
> >    to achieve scaling and redundancy in services.
> The current bottleneck is the rest api from core. IMO bulk actions, 
> filtering and sorting should be added there before trying to have 
> multiple postorius/hyperkitty instances serving the same core...

I didn't say they will be served from the same core ;-)

Yes, the Core's API can be further improved to be more efficient. But
that
doesn't prevent us from scaling. I understand there are issues around
multiple
instances of containers, but I have ideas to get around most of them.


-- 
  Abhilash Raj
  maxking at asynchronous.in