[Borgbackup] Problems after borg client upgrade

[Gareth Evans]

I use borg client 1.2.4 on Debian bookworm with 1.2.7 on a FreeBSD server, of which (the server) I am not in control.

Repos are encrypted using local keyfiles, and were created quite some time ago.

On the first day of the month, the last-day-of-last-month's repo is converted to that month's archive for the year.  I have a script which does:

borg delete redacted at redacted:$repo::m$lastmonth
borg rename redacted at redacted:$repo::d$yesterday m$lastmonth

$ borg --version
borg 1.2.4
$ sudo borg check redacted at redacted:etc
$

After upgrading the client version to 1.2.7 from bookworm-backports, problems are reported with the archives which have been renamed.

--
$ sudo borg check redacted at redacted:etc
Archive TAM authentication issue for archive m07: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
Archive TAM authentication issue for archive m08: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
Archive TAM authentication issue for archive m09: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
Archive TAM authentication issue for archive m10: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
Archive TAM authentication issue for archive m11: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
Archive TAM authentication issue for archive m12: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
Archive TAM authentication issue for archive m01: Data integrity error: Archive authentication did not verify
This archive will be *removed* from the manifest! It will be deleted.
162 orphaned objects found!
Archive consistency check complete, problems found.
--

Listing, for example, m01 produces

--
$ sudo borg list redacted at redacted:etc::m01
Data integrity error: Archive authentication did not verify
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5343, in main
    exit_code = archiver.run(args)
                ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 5263, in run
    return set_ec(func(args))
                  ^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 189, in wrapper
    return method(self, args, repository=repository, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1385, in do_list
    return self._list_archive(args, repository, manifest, key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1414, in _list_archive
    _list_inner(cache=None)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1402, in _list_inner
    archive = Archive(repository, key, manifest, args.location.archive, cache=cache,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 488, in __init__
    self.load(info.id)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 501, in load
    self.metadata = self._load_meta(self.id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 493, in _load_meta
    archive, self.tam_verified, _ = self.key.unpack_and_verify_archive(data, force_tam_not_required=True)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/borg/crypto/key.py", line 329, in unpack_and_verify_archive
    raise ArchiveTAMInvalid()
borg.crypto.key.ArchiveTAMInvalid: Data integrity error: Archive authentication did not verify

Platform: Linux qwerty 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64
Linux: Unknown Linux  
Borg: 1.2.7  Python: CPython 3.11.2 msgpack: 1.0.3 fuse: pyfuse3 3.2.1 [pyfuse3,llfuse]
PID: 1369927  CWD: /home/user
sys.argv: ['/usr/bin/borg', 'list', 'redacted at redacted:etc::m01']
SSH_ORIGINAL_COMMAND: None
--

It is still possible to create new repos, which check OK, but not to list the affected repos due to errors like the above.

Is this a bug?

I can't see anything relevant in the change log

https://borgbackup.readthedocs.io/en/stable/changes.html#upgrade-notes

I wonder if a borg upgrade would help, but I can't see a way to upgrade a remote keyfile-encrypted repo.

$ sudo ssh redacted at redacted borg upgrade -v etc
making a hardlink copy in /data1/home/redacted/etc.before-upgrade-2024-02-25-22:22:35
opening attic repository with borg and converting
no key file found for repository
converting repo index /data1/home/redacted/etc/index.91850
converting 74 segments...
converting borg 0.xx to borg current
no key file found for repository
$

This appears to fail, as the same output appears if the command is repeated - I presume because the keyfile is not provided - how can it be?

The problems disappear if borg client is reverted to 1.2.4

Any ideas what has happened or how to fix it would be much appreciated.

Thanks,
Gareth