summary of my last hacking streak

Sat Oct 17 02:47:09 EDT 2015

Hi,

This week, I took my laptop out to the countryside, without internet
(but with a HAM radio ;), to focus on peace and quiet and maybe work on
borg.

First off, I must apologise for the mess I created on the master
branch. I attempted to create a master branch in my own fork with all
the branches merged in and I mistakenly pushed that to the main repo,
which closed all the pull requests and put all my code live at once. I
have reverted the change and now only the --version pull request is
there.

I am at a loss at words at how to handle this... I guess I could rebase
those branches on top of the new messed up master, or we could reset the
master branch again. I am not sure what is the best way to proceed, but
I can't fix this cleanly myself, as I cannot force-push to master. :(

Again, apologies. Here is the rest of the mail i was hoping to send
happily an hour ago...

Pull requests
=============

I ended up with 6 feature branchs which i submitted as pull requests:

* i18n: translation support for borg
  https://github.com/borgbackup/borg/pull/287
* man-builder: failed attempt at writing a manpage formatter for
  argparse - the latter is such a tangled mess that i can't make head
  or tails of it, and i gave up: let's wait for click
  https://github.com/borgbackup/borg/issues/208
* no-inplace: operate on a "copy" during upgrades, the previous
  "inplace" mode is still available through `--inplace`
  https://github.com/borgbackup/borg/pull/280
* upgrader-index-fixes: fixes for the upgrader index problems, not
  sure how to create a unit test for this one, as repository.check()
  doesn't find the error (only create does)
  https://github.com/borgbackup/borg/pull/281
* verbosity: lots of tweaks to verbosity levels, --progress and
  --stats
  https://github.com/borgbackup/borg/pull/288
* version-arg: a --version flag! (this was mistakenly pushed directly to
  master, sorry, let me know if i need to revert)
* x-option, add -x to --do-not-cross-mountpoints setting
  https://github.com/borgbackup/borg/pull/282

Some of those conflict with each other: i18n and verbosity touch similar
areas, but the merge is fairly trivial. I have merged all the above (but
man-builder) into my master branch:

https://github.com/anarcat/borg/commits/master

(Update: this was mistakenly pushed to the main master branch, and then
reverted.)

Bug reports
===========

* tox seems to run the Attic test suite:
  https://github.com/borgbackup/borg/issues/283
* can't run tests offline: https://github.com/borgbackup/borg/issues/284
* lock.exclusive left behind:
  https://github.com/borgbackup/borg/issues/285

Other notes
===========

Upgrade notes
-------------

I ran a new conversion from Attic, this time on a fresh backup of my
laptop (i usually work on my workstation). The backup looked like:

    # attic create --exclude-caches -e /home/anarcat/mp3/ -e /home/pbuilder/ -e /home/anarcat/video/ -e /home/anarcat/iso/ -e  /home/anarcat/books -e /home/anarcat/books-incoming -e .cache --stats --do-not-cross-mountpoints /media/anarcat/calyx/attic-angela::2015-10-15-first /
    Initializing cache...
    ------------------------------------------------------------------------------
    Archive name: 2015-10-15-first
    Archive fingerprint: 5626925d54fae692e7ca5cb852e6629661d4ac5de00ad6b6494979980ec7822c
    Start time: Thu Oct 15 12:03:34 2015
    End time: Thu Oct 15 15:28:44 2015
    Duration: 3 hours 25 minutes 10.06 seconds
    Number of files: 847008

                           Original size      Compressed size    Deduplicated size
    This archive:               56.78 GB             38.82 GB             35.55 GB
    All archives:               56.78 GB             38.82 GB             35.55 GB
    ------------------------------------------------------------------------------

First borg run after upgrade was fast enough:

    ------------------------------------------------------------------------------
    Archive name: 2015-10-15-post-attic
    Archive fingerprint: 44812971c502a25dcad61a7664eab3559405c6f1fd8b38bc50118f4619dbca72
    Start time: Thu Oct 15 18:36:02 2015
    End time: Thu Oct 15 18:49:15 2015
    Duration: 13 minutes 13.69 seconds
    Number of files: 847084

                           Original size      Compressed size    Deduplicated size
    This archive:               56.78 GB             38.85 GB             41.12 MB
    All archives:              113.57 GB             77.67 GB             35.59 GB

                           Unique chunks         Total chunks
    Chunk index:                 1430830              3255749
    ------------------------------------------------------------------------------

So it seems the chunks cache got reused properly this time, maybe
because the dataset is smaller.

weird integrity failure
-----------------------

When running tests with `-s`, we see some exceptions show up in the
log. It's a little confusing because we have the feeling that tests
fail because of the exception, while it seems the error is normal.

Failing test sample:

    borg/testsuite/archiver.py::ArchiverCheckTestCase::test_missing_archive_item_chunk Exception ignored in: <bound method Repository.__del__ of <Repository /tmp/tmpq8mq3loc/repository>>
    Traceback (most recent call last):
      File "/home/anarcat/src/borg/borg/repository.py", line 69, in __del__
        self.close()
      File "/home/anarcat/src/borg/borg/repository.py", line 155, in close
        self.lock.release()
      File "/home/anarcat/src/borg/borg/locking.py", line 281, in release
        self._roster.modify(EXCLUSIVE, REMOVE)
      File "/home/anarcat/src/borg/borg/locking.py", line 203, in modify
        elements.remove(self.id)
    KeyError: (('angela', 28267, 2564556544),)
    PASSED

Another:

    borg/testsuite/archiver.py::RemoteArchiverTestCase::test_corrupted_repository Segment entry checksum mismatch [segment 2, offset 8]
    Index object count mismatch. 147 != 0
    PASSED

Maybe those warnings are unimportant and should be ignored, but they
certainly make tests more confusing, and if those errors are expected,
maybe they shouldn't output those errors?

argv
----

It would be nice if borg would show up as "borg" instead of "python
borg" or whatever. bup does this through unpythonize_argv() which does
some pretty heavy memset() stuff to clear it up. there has to be a
better way (tm).

See: https://github.com/borgbackup/borg/issues/286

verbosity and logging
---------------------

it's harder to censor the unchanged files with stderr output: we
basically need to shove it back to stdout to grep it. that is
annoying.

basically, -v is way too verbose - it's unusable. also, as it is,
--progress doesn't show *any* progress until the file cache is loaded,
which is confusing, as we don't know if borg is waiting for a lock,
blocked or what.

the basic problem could be that --progress conflicts with the per-file
output. to resolve that, i have pushed the file listing down to the
DEBUG level for now, but the question of how to handle different
verbosity levels still stands

but i do feel confident now that the logging-refactor branch can be
merged in, and in fact most branches i have been working on have been
based on it...

oh, and --progress now knows about the terminal width, so it makes good
use of all the space to show file paths. it also shows the number of
files found so far.

--stats could similarly be improved as well...

format_file_sizes
-----------------

This could be reimplemented to cover more than terabytes, there's some
neat code from stackexchange for this...

http://stackoverflow.com/a/1094933/1174784

StableDict
----------

Why is that thing necessary, when we have OrderedDict?