[Borgbackup] faster / better deletion, for a bounty?

[Mario Emmenlauer]

Dear Borg developers,

thanks for the awesome tool! I've been using borgbackup now for
more than 6 months and created 533 backups successfully!

But now the disk is ~98% full and suddenly I have some troubles :-)
I've received great feedback already in IRC but some questions still
trouble me:
(1) My archive is now 3.4 TB (reported with 'du'), but borg list says
    the deduplicated archive size is 1.82 TB. Why are the two numbers
    off by 50%? Below the full output of my borg list.

(2) In the last months, my backup size went up quite a lot, even though
    I did not change anything in borg. So I'd like to reverse engineer
    which archives (or which files) contribute to the sudden increase in
    size. I tried "borg list" on all archives, but only 7 have ~3 GB of
    deduplicated space, and all others have less than 1 GB of dedup space!
    I assumed 533 archives of ~1 GB dedup size = 533 GB total, but my
    math must be quite wrong? I saw the documentation of "borg list"
    but it does not help me understand :-(
    How would I find the archives that free most space when deleted?

(3) borg delete was incredibly slow for me. I killed it after two hours,
    and it had read 500GB of the archive by then (reported with iotop).
    I understood from IRC discussion that both prune and delete would
    require reading the full 3.4 TB once per run, to sanitize some index?
    That would break borg usage for me, since this will very much wear
    the disk and also takes ~8hrs on my encrypted drive! Am I doing some-
    thing wrong? Are there tricks or workarounds, for example when
    deleting only from localhost?
    I'd like to offer a bounty of ~€20-€25 for a better solution, or a
    generally much faster delete and/or much faster prune. If possible
    I'd rather not have borg read the full 3.4TB archive!

PS: My preferred deletion pattern would keep an increasing number of
    archives over time, like monthly backups from the past 10 years,
    weekly from the past year, and  daily from past month. I can build
    this list of deletions with bash easily! But borg delete or prune
    are currently *way* to slow to be used this way :-(


#> borg list archive::somebackup
Number of files: 1796064
                       Original size      Compressed size    Deduplicated size
This archive:               95.27 GB             70.53 GB            178.00 MB
All archives:               78.26 TB             65.13 TB              1.82 TB
                       Unique chunks         Total chunks
Chunk index:                 9733154            414693364


Thanks a lot and all the best,

    Mario Emmenlauer


--
BioDataAnalysis GmbH, Mario Emmenlauer      Tel. Buero: +49-89-74677203
Balanstr. 43                   mailto: memmenlauer * biodataanalysis.de
D-81669 München                          http://www.biodataanalysis.de/