[scikit-image] A request for more frequent releases

[Mark Harfouche]

Hi Juan,

Thank you for the quick reply.

The amount of work to overhaul the release infrastructure is definitely non
trivial. I totally agree. That is why it isn't worth doing
Changing the build infrastructure and the release infrastructure needs to
be done carefully (as I have learned a few times asking others to install
empty packages of mine :S).
That said, I think we can approach your "tag to release" idea, especially
with the amount of resources available for OSS on Github.

You are totally, rebasing is really an advanced feature and quite
difficult. I've had trouble explaining it to many of my team mates.
Lets keep discussing ideas for the changelog. I definitely think we can
improve on the current state of "go through the old PRs".

A monthly release cycle is fine, but as is, I think it is too much work to
expect from people "donating" their time. That said, I'm not the one
cutting the releases.

I know the build infrastructure is complex, but this isn't something that
was ever easy to understand. Bots aren't easy to understand either.
This is because PyPi wants to target the oldest linux it can, and not
because scikit-image is much harder to compile than other python projects.

As it stands, we were surprised by failures on 32 bit linux when 0.14 came
around. This should have been caught and marked as `xfail` a long time ago.
By having the wheel infrastructure built into the every day process, we
should be much more confident that we could release 0.15 "tomorrow" without
fear that something would break on one of our supported architectures.
And yes, I mean .15 since I've been living on master for the last few
months, though I've slowly been reverting to 0.14 + monkey patching (which
I really don't like).

I posted it here as it seems that these organizational changes are
discussed more on here than on github.

Mark

On Sun, Sep 9, 2018 at 1:58 AM Juan Nunez-Iglesias <jni at fastmail.com> wrote:

> Hi Mark!
>
> Coincidentally I'm cutting 0.14.1 right now!
>
> Actually Stéfan already suggested this a while back. It hasn't happened
> not because of any actual opposition to the idea, but because all of the
> automation work you suggest is not trivial. I'd also like to bring up
> François's point that our build machinery is already quite complex, and it
> can be somewhat of an impediment to newcomers.
>
> On the other hand, I actually believe that we can *both* simplify the
> build machinery *and* get rolling releases, but this is not an afternoon's
> fix.
>
> The only part of your proposal that I find objectionable is calling
> constant rebases a feature. Rebasing is an extremely unfriendly workflow
> for beginners.
>
> My proposed amendment to your proposal is that we move to a monthly cycle,
> and create a "release bot" that creates a PR on the 1st of each month with
> the contribs.py output. Someone can modify the PR to tweak the output where
> needed, merge, and tag.
>
> Lastly, I think this would be a good comment to add to our roadmap RFC:
> https://github.com/scikit-image/scikit-image/issues/3263
>
> Juan.
>
>
> On Sun, Sep 9, 2018, at 2:52 PM, Mark Harfouche wrote:
>
> Hi All,
>
> I would like to request that scikit-image work toward a much more frequent
> release cycle.
> Let me give some anecdotal reasons why this might be a good idea.
>
> As a new contributor, scikit-image has given me a few things:
> 1. It has taught me a lot about some best practices with respect to
> software development.
> 2. It has given me "free" reviews of software and features I would like to
> implement with respect to image processing.
> 3. It has taught me a lot about releasing software in a systematic manner.
> 4. Once code that I depend on is included in scikit-image, it likely gets
> tests and maintained much better than code I would write on my own. Lately,
> two good examples have been Python 3.7 and numpy 1.15 compatibility.
>
> Of course, now that I know a little more about collaborative software
> development, I try to give back.
>
> Unfortunately, as you all know, contributing doesn't come for free.
>
> With the long time lag between releases, writing code or bug fixes for
> scikit-image means that I have to replicate them for my own projects.
> My current workaround has converged to having a subdirectory
> `upstream/skimage` where I hold a few bug fixes that I have come to rely on
> from the `master` branch of `scikit-image`.
> Not only is that costly, but refactoring my code after scikit-image will
> get a release will also be costly. As such, I now have a disincentive to
> submit patches, bug reports, speedups to scikit-image.
> Having my own `upstream/skimage` directory means that I will likely
> diverge from scikit-image's best practices when I implement my own
> functions.
>
> This is going to make it harder and harder to find time to contribute back
> to scikit-image as the complexity of my projects grows. Not contributing to
> scikit-image will come at some cost to me, but I would like to emphasize
> that I am likely not the only one that is probably facing these dilemmas.
> This means that whatever code I or others like me develop, be it for
> speeding up warping, or image registration on cuda (to name two features I
> am working on), will likely not make it to scikit-image if the cost of
> contributing stays high.
>
> Currently, asking for a 2-week release cycle is guaranteed to be rejected.
>
> The Release guidelines are 125 lines long (if we ignore the Debian and
> conda-forge instructions).
> https://github.com/scikit-image/scikit-image/blob/master/RELEASE.txt
>
>   - Many files need to be renamed. This can be done on an automated script.
>   - Docs need to be rebuilt and uploaded. We can work to do this with
> travis deploy, or readthedocs
>   - Release notes need to be manually scanned. We can ask for this to be
> done in each PR.
>   - Updating __version__ is a pain. I really suggest we work toward using
> versioneer and tags instead of branches for releases.
>   - Building wheels is done in separate repo. I really suggest we merge
> scikit-image-wheels into scikit-image itself and build wheels on every PR.
>   - conda-forge now has automated scripts that will bump the version
> number. This should be moved out as these instructions are stale and
> available on the feedstock itself.
>   - Debian can release however often they like.
>
> Asking for release notes to be updated in PRs will cause conflicts in PRs.
> I think this is a feature, not a bug. Many times tests are failing because
> of typos in small changes in the code. Asking for an updated rebase will
> help us ensure real time code correctness.
>
> I am more than glad to help work on any of the points I mentioned above
> and any others you find necessary.
>
> I know this will take work, but I think having much more frequent
> releases, timed releases, will help ensure that people can rely on their
> contributions making it to PyPi or conda-forge so that they can ask others
> to rely on them.
>
> Thank you very much for considering this,
>
> Mark
>
>
> *_______________________________________________*
> scikit-image mailing list
> scikit-image at python.org
> https://mail.python.org/mailman/listinfo/scikit-image
>
>
> _______________________________________________
> scikit-image mailing list
> scikit-image at python.org
> https://mail.python.org/mailman/listinfo/scikit-image
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/scikit-image/attachments/20180909/08e5274d/attachment.html>