[Borgbackup] bundling msgpack for borgback distribution

[Felix Schwarz]

Hi Thomas,

thank you very much for your response.

I built a RPM with a bundled version of msgpack 0.5.6 (which is the current
version in Fedora 29 anyway).

Am 07.05.19 um 15:26 schrieb Thomas Waldmann:
> borg internally uses msgpack quite a lot, so any major breakage would
> blow up things quite spectacularly.

Tests were run with

   py.test-3 -x -vk "not test_non_ascii_acl and not test_fuse and not
benchmark and not test_dash_open and not test_mount_hardlinks"
$PYTHONPATH/borg/testsuite/*.py

and everything passed so I hope everything is fine.

One thing I noticed was that I got test failures when running 1.1.9 with a
pure-python msgpack. That costed me a bit of time until I noticed this seems
to be a problem in upstream borg (not my initial RPM packaging).

Are you interested in a bug report? (I don't care too much now as I'm looking
forward to 1.2).

> What I was wondering about:
> 
> Is there no process in linux distributions that holds maintainers back
> from upgrading to new package versions, if the upgrade breaks existing
> and well-declared dependencies?

Short answer "no".
Basically the current breakage seems to be a volunteer/maintainer problem:

There is a maintainer for python-msgpack who wants to provide the latest
msgpack which should fix a security issue. I guess this is not relevant for
borgbackup as it does not work on untrusted data. However there are also other
dependent packages in Fedora (+ user code which might use python-msgpack)
which might be affected.

On the other side borgbackup in Fedora is really only maintained by a single
volunteer. The breakage was spotted early on (in January) but for whatever
reason no action was taken for the Fedora borgbackup package.

As I was not doing "maintainer stuff" for borgbackup in years I did not
prioritize the bug until F30 was published and it became clear to me that
nobody but me was willing to invest time in that.

Yes, Fedora's tooling is inadequate for that - allowing maintainers to push
packages which break the upgrade path or other packages. However in this
situation it would have probably resulted in borgbackup being removed
altogether from Fedora 30.

> Assuming you must keep the "too new" msgpack package, bundling is maybe
> the best option - please give feedback about that on our issue tracker,
> so maybe we could implement it upstream or at least other package
> maintainers can find it there.

I'm attaching my patches here - not sure if they really help upstream.
1. I patched python-msgpack to use only relative imports. That way I can move
   the msgpack sources into src/borg/_msgpack (Python package
   "borg._msgpack").
   I don't know why upstream only uses absolute imports but that part might
   actually go upstream.
2. Adding the two Cython files to borg's setup.py. msgpack uses Cython's C++
   compiler so I did the same.
3. Patching borg to use msgpack from borg._msgpack - that patch would be much
   smaller with borg 1.2 as you added a msgpack "shim" module.

So if you like to supporting bundling inside of borgbackup I could try finding
time to refine my patch #2.

Felix
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0010-msgpack-use-relative-imports.patch
Type: text/x-patch
Size: 3021 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190510/37b24177/attachment-0003.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0011-also-build-msgpack.patch
Type: text/x-patch
Size: 4234 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190510/37b24177/attachment-0004.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0012-use-bundled-msgpack.patch
Type: text/x-patch
Size: 5446 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20190510/37b24177/attachment-0005.bin>