[Borgbackup] Central backup and ~/.cache/borg/ piling up

[Marian Beermann]

>> Since --no-cache-sync is marked experimental, I want to ask, what
>> other downsides one can expect, and what experiences others have had
>> using these flags, or otherwise tackling the issue in question.
> 
> IIUC the doc, each and every backup of your's is re-processed (almost?)
> the same as if it was the first one - this doesn't look optimal at all;
> so, using these options should be reduced to very special cases (I guess
> Thomas could enlighten us about which they could be).
> 
> In a word, you lose (very much) time and bloat the network for a minor
> sparing - not to mention that if you have a lot of machines to backup,
> the whole thing is taking much too long to achieve.
> 
> So, IF my doc interpretation's correct, you lose dozens of time whatever
> you hope to gain using --no-cache-sync, namely time and network
> availability.

I originally wrote the code for --no-cache-sync a few years ago, but I
didn't remember what it does, so I re-read the code [1] to refresh my
memory :)

--no-cache-sync uses the local cache *if* it is in sync. Otherwise it
downloads a list of all chunks in the repository (that's about "Unique
chunks" * 32 bytes in network traffic, in your case 31886048 * 32 bytes
= 1 GB!) and uses that for deduplication. Because it only has the the ID
of chunks, but doesn't know their size or compressed size, the stats
will be all wrong (all deduped chunks will show up with a size of zero
bytes). This might result in some situations (archives being deleted
etc.) in a cache sync (--possibly on another host) to download actual
data chunks to figure their (uncompressed) size out.

IIRC we added it to see if it would work in practice or not. I dunno if
someone tried it in a bigger scenario and what the results there might
have been.

-Marian

[1]
https://github.com/borgbackup/borg/commit/8aa745ddbd8ee1cddb3374673eb3eb08a9d5a8da