[Borgbackup] bundling msgpack for borgback distribution

[Thomas Waldmann]

> how comprehensive is borgbackup's test suite when it comes to msgpack?

Good question.

borg internally uses msgpack quite a lot, so any major breakage would
blow up things quite spectacularly.

otoh, the test suite is not made to somehow test for the slightly
incompatible changes in the msgpack api - borg expects to be run with
the msgpack versions as required in setup.py and it always installs
correct versions when using pip / tox / etc.

> If the test suite does not report any errors may I assume that the used
> version of msgpack is likely working correctly?

For the existing msgpack versions, I guess this could be found out be
digging through issue tracker (incl. closed issues) and mailing list and
see if all the problems encountered also will be encountered when
running the tests.

You can't really be sure for future msgpack changes, though.

For borg 1.2, I wrapped the whole msgpack api to avoid such pain in
future (or at least be able to more quickly / easily adapt to it), but
that doesn't help us for 1.1.

> Unfortunately Fedora 30 ships msgpack 0.6.1 but there is no released version
> of borgbackup which can handle this version of msgpack.

What I was wondering about:

Is there no process in linux distributions that holds maintainers back
from upgrading to new package versions, if the upgrade breaks existing
and well-declared dependencies?

Also, is there something else actually requiring msgpack >= 0.6.0 so the
upgrade was needed for that?

Or is it just for the fun of having the latest version? :)

> I assume the simplest way to achieve this is to bundle msgpack 0.5.6 into
> borgbackup for distribution.

Sounds good (assuming borg actually only loads the bundled code and not
the system wide one, if also installed).

Make sure the bundled stuff also has the shared libs compiled from
msgpack C code or borg will be slow and emit a "using slow msgpack"
warning, if it falls back to pure python msgpack code.

borg 1.1.9 release has a runtime check for the msgpack version and exits
if it does not meet requirements.

> During packaging we run borg's test suite.
> Assuming the test suite passes may I assume that borg will be fine-ish?

It's better than not running the tests, but I am not totally sure it is
sufficient.

> (I know that is a hack and I don't like doing that to such an important
> package like borg. But on the other hand borg is unusable on Fedora 30 right
> now. The only thing which could be worse is if borg starts eating the users
> backup data.)

Assuming you must keep the "too new" msgpack package, bundling is maybe
the best option - please give feedback about that on our issue tracker,
so maybe we could implement it upstream or at least other package
maintainers can find it there.

Another option (for users, independent of the distribution) is to use
our "fat binary" from github releases page.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393