[Borgbackup] borg 1.0.3 crashes on prune

[Thomas Waldmann]

>> $ borg prune -v --list --keep-daily=7 --keep-weekly=4
>> --keep-monthly=12 --prefix server-vbox /backup/borg
>> ...
>> Object with key
>> b'\x91\xd9\xd0\xde\xad\xfaz\xeeEL\xef\x80\x1c?\xfeX"\xcf\xa8\xfe2\x14\xec\xe4\x89\xacy\x97}~\xf1y'
>> not found in repository /backup/borg.

The object seems to be not in the repo.

>> Traceback (most recent call last):
>>   File "/usr/lib64/python3.4/site-packages/borg/archiver.py", line
>> 610, in do_prune
>>     Archive(repository, key, manifest, archive.name, cache).delete(stats)
>>   File "/usr/lib64/python3.4/site-packages/borg/archive.py", line 450,
>> in delete
>>     self.cache.chunk_decref(chunk_id, stats)
>>   File "/usr/lib64/python3.4/site-packages/borg/cache.py", line 392,
>> in chunk_decref
>>     self.repository.delete(id, wait=False)
>>   File "/usr/lib64/python3.4/site-packages/borg/repository.py", line
>> 476, in delete
>>     raise self.ObjectNotFound(id, self.path) from None
>> borg.repository.ObjectNotFound:
>> (b'\x91\xd9\xd0\xde\xad\xfaz\xeeEL\xef\x80\x1c?\xfeX"\xcf\xa8\xfe2\x14\xec\xe4\x89\xacy\x97}~\xf1y',
>> '/backup/borg')

Strange, that (obviously) should not happen.

It decremented the reference counter in the chunks index, found it 0,
then (successfully) deleted the entry from the chunks index, then tried
to delete the (now unused) object from the repo, but it was not there -
boom.

So, it looks like the index and repo contents did not agree.

Can it be that 2 borgs accidentally wrote to the repo in parallel
because you manually broke the lock (borg break-lock)?

Were there any issues before this one when accessing the backup repo?

>> '/backup/borg']

Is that the correct repo path? Is it a local device or via network
somehow? How?

If via network: did you have network interruptions while writing to the
repo?

borg check -v might be a good idea now. check its result.

>> How can I recover without endangering my data?

if there is a problem and the repo is valuable, make a copy of it before
running borg check --repair on it.

if you are super paranoid, you can also run borg extract --dry-run on
some or all archives afterwards. takes time, but is best assertion of
archive health.

>> Crashing backup software makes me nervous.

Me, too, it would be good if we could find the root cause of this.


In this special case, the failing operation that caused the traceback is
not the big problem - it wanted to delete something that was already gone.

But a chunk reference counter (from chunk index) disagreeing with repo
state is the real problem here.

-- 

GPG ID: FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393