[Borgbackup] conversion from attic to borg

[Antoine Beaupré]

On 2015-12-04 14:02:07, Dan Christensen wrote:
> Antoine Beaupré <anarcat at debian.org> writes:
>
>> On 2015-12-04 09:08:55, Dan Christensen wrote:
>>
>>> Also, how fast is it, with and without the --inplace option?
>>
>> It's pretty fast! `--inplace` just skips the step where it copies
>> (equivalent of `cp -al`) the repository into a backup repo. Once that
>> copy is done, it just needs to write a few bytes to every segment, which
>> takes at most a few minutes.
>
> It looks like the upgrade processes is changing the files by seeking
> into them and writing new bytes.  This doesn't break hardlinks, so the
> "backup" copy that is made is also upgraded and is unusable by attic.

I *believe* the open/writing process deliberately breaks
hardlinks. There's a unit test I wrote to confirm this as well:

https://github.com/borgbackup/borg/blob/master/borg/testsuite/upgrader.py#L188

> But if just about every file in the repo needs to be changed, what point
> is there in making hardlinks?

The point is that we get meaningful progress information. The upgrade
command was originally written as "inplace only", that is: it would
change all the files and assume *you* would perform the backup
first, with `cp -a` or `cp -al` or similar. Then i did a bit more
hand-holding and thought of doing the copy as part of the upgrade
command directly. The problem with doing that copy then is that it's
harder to get an idea of progress, as we copy what seems to be an
arbitrary hierarchy of files recursively, and we don't know when that
long process finishes.

By making a forest of hardlinks and rewriting the files one by one, we
effectively do a copy, but we have progress information along the way.

> The backup and the upgraded repo aren't going to be able to share
> files, so borg will have to either make copies at the start or break
> the hardlinks by making new copies of the files during the upgrade.
> It seems cleanest to just make a true copy of the repo at the start.

You can still do that: do the "true copy" yourself and use `--inplace`.

> (The only advantage of the hardlinks would be if there are substantial
> files that can be shared between the repos, but I don't think that's
> the case.)
>
> https://github.com/borgbackup/borg/issues/466

Let's move the conversation to that issue: if the hardlinks are not
broken, then this is indeed a bug. I hope the rationale for the
hardlinks was made clearer in the above comment, if not, i'd be happy to
clarify further.

> Antoine Beaupré <anarcat at debian.org> writes:
>
>> I guess you *could* run the upgrade process multiple times, once per
>> client. The server side is "idempotent", that is: it can be run
>> multiple times without ill effects, and it will skip work already
>> done.
>
> But since remote operation isn't yet supported, this is a moot point.
> I'll just run the upgrade on the backup server, and then each client
> will have to rebuild their cache the first time.

True.

> I guess there could be a "--files-cache-only" option to upgrade
> for this use case.

That would make sense. There's a new "ToggleAction" option that was
added for --progress vs --no-progress recently, maybe we could expand
that to have topical flags like --{no-,}files-cache --{no,}segments and
so on...

>> I don't think `--no-files-cache` is a valid option for the upgrade
>> command, but it could be added!
>
> Ah, it is shown in the usage information:
>
> usage: borg-linux64 upgrade [-h] [-v] [--show-rc] [--no-files-cache]
>                             [--umask M] [--remote-path PATH] [-n] [-i]
>                             [REPOSITORY]
>
> It looks like some of the other things shown might not apply either.

Yet another bug i guess - we should obey --no-files-cache in upgrade.

A.

-- 
Quis custodiet ipsos custodes?
Who watches the watchmen?
Qui police la police?
Tu. You. Toi.