[Borgbackup] purge not deleting data?
Adrian Klaver
adrian.klaver at aklaver.com
Mon Jan 23 09:59:35 EST 2017
On 01/23/2017 03:39 AM, devzero at web.de wrote:
> Hello,
>
> i run a rsync based daily backup where a number of hosts files are being staged to some local filesystem (one subdir per host) and then put into individual borg-repo on a "one-dir-per-host" basis (see script below)
>
> every few days, i see that borg prune does not appear to purge data from the repos, and for my curiosity it happens for all repos at the same day/backup-run - though rsync tells it did delete files from the rsync copy.
>
> so i wonder under which circumstances borg prune skips deleting files !?
>
> as the repos were created at different points in time i cannot explain this to me, maybe someone has a clue how this can be explained or analyzed?
Assuming you are referring to the lines below where Deleted data = 0, then the archives on
8/1/2017 and 15/1/2017 represent the last one of each week and would be saved
by the --keep-weekly 8. Not sure about the 14/1/2017 one as that should fall
within the --keep-daily 14, unless there was another later archive on that day.
How did you determine the Deleted data for each archive?
Also I to back Thomas's suggestion, I would use y-m-d dates. You do use that in the
rsync output and it would make comparing things easier.
Re: the suggestion to use --list, it gives you the day of the week which helps me with
seeing what pruning is doing:
-----------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 1.33 GB 1.31 GB 89.14 MB
All archives: 25.90 GB 24.36 GB 4.15 GB
Unique chunks Total chunks
Chunk index: 37785 516017
------------------------------------------------------------------------------
Keeping archive: tito-012217_1900 Sun, 2017-01-22 19:00:03
Keeping archive: tito-012117_1900 Sat, 2017-01-21 19:00:03
Keeping archive: tito-012017_1900 Fri, 2017-01-20 19:00:03
Keeping archive: tito-011917_1900 Thu, 2017-01-19 19:00:03
Keeping archive: tito-011817_1900 Wed, 2017-01-18 19:00:04
Keeping archive: tito-011717_1900 Tue, 2017-01-17 19:00:04
Keeping archive: tito-011617_1900 Mon, 2017-01-16 19:00:04
Keeping archive: tito-011517_1900 Sun, 2017-01-15 19:00:04
Keeping archive: tito-010817_1900 Sun, 2017-01-08 19:00:04
Keeping archive: tito-010117_1900 Sun, 2017-01-01 19:00:05
Keeping archive: tito-123116_1900 Sat, 2016-12-31 19:00:05
Keeping archive: tito-122516_1900 Sun, 2016-12-25 19:00:04
Keeping archive: tito-121816_1900 Sun, 2016-12-18 19:00:03
Pruning archive: tito-011417_1900 Sat, 2017-01-14 19:00:04
>
> regards
> roland
>
>
> [root at backupvm2]# egrep -Hi "This archive|Deleted" borg_*-*-2017*.err
> borg_04-01-2017_03-45.err:This archive: 60.42 GB 45.51 GB 536.55 MB
> borg_04-01-2017_03-45.err:Deleted data: -99.61 GB -82.57 GB -1.32 GB
> borg_05-01-2017_04-11.err:This archive: 60.39 GB 45.49 GB 1.17 GB
> borg_05-01-2017_04-11.err:Deleted data: -60.54 GB -45.61 GB -494.64 MB
> borg_06-01-2017_03-32.err:This archive: 60.37 GB 45.46 GB 1.12 GB
> borg_06-01-2017_03-32.err:Deleted data: -60.55 GB -45.62 GB -985.93 MB
> borg_07-01-2017_05-00.err:This archive: 60.40 GB 45.58 GB 1.32 GB
> borg_07-01-2017_05-00.err:Deleted data: -60.55 GB -45.63 GB -499.40 MB
> borg_08-01-2017_03-10.err:This archive: 60.40 GB 45.58 GB 536.57 MB
> borg_08-01-2017_03-10.err:Deleted data: 0 B 0 B 0 B
> borg_09-01-2017_03-24.err:This archive: 60.39 GB 45.58 GB 589.20 MB
> borg_09-01-2017_03-24.err:Deleted data: -60.54 GB -45.63 GB -473.20 MB
> borg_10-01-2017_03-34.err:This archive: 60.39 GB 45.59 GB 1.11 GB
> borg_10-01-2017_03-34.err:Deleted data: -60.54 GB -45.63 GB -511.70 MB
> borg_11-01-2017_03-51.err:This archive: 60.46 GB 45.63 GB 910.10 MB
> borg_11-01-2017_03-51.err:Deleted data: -60.54 GB -45.63 GB -508.64 MB
> borg_12-01-2017_03-51.err:This archive: 60.46 GB 45.64 GB 1.33 GB
> borg_12-01-2017_03-51.err:Deleted data: -60.54 GB -45.63 GB -507.77 MB
> borg_13-01-2017_04-50.err:This archive: 60.55 GB 45.71 GB 1.50 GB
> borg_13-01-2017_04-50.err:Deleted data: -60.55 GB -45.63 GB -513.74 MB
> borg_14-01-2017_04-32.err:This archive: 60.56 GB 45.71 GB 926.04 MB
> borg_14-01-2017_04-32.err:Deleted data: 0 B 0 B 0 B
> borg_15-01-2017_03-14.err:This archive: 60.54 GB 45.71 GB 565.45 MB
> borg_15-01-2017_03-14.err:Deleted data: 0 B 0 B 0 B
> borg_16-01-2017_03-49.err:This archive: 60.54 GB 45.71 GB 529.96 MB
> borg_16-01-2017_03-49.err:Deleted data: -60.46 GB -45.56 GB -512.22 MB
> borg_17-01-2017_04-14.err:This archive: 60.56 GB 45.72 GB 1.57 GB
> borg_17-01-2017_04-14.err:Deleted data: -60.44 GB -45.53 GB -358.81 MB
>
> ls -1 */rsync.log |while read line;do echo $line $(grep -i deleting $line|wc -l);done
> 2017-01-03-2303/rsync.log 599
> 2017-01-04-2303/rsync.log 622
> 2017-01-05-2303/rsync.log 2525
> 2017-01-06-2303/rsync.log 711
> 2017-01-07-2303/rsync.log 6305
> 2017-01-08-2303/rsync.log 581
> 2017-01-09-2303/rsync.log 488
> 2017-01-10-2303/rsync.log 1605
> 2017-01-11-2303/rsync.log 738
> 2017-01-12-2303/rsync.log 1669
> 2017-01-13-2303/rsync.log 5394
> 2017-01-14-2303/rsync.log 5056
> 2017-01-15-2303/rsync.log 478
> 2017-01-16-2303/rsync.log 487
>
>
> [root at backupvm2 backup]# cat borg_backup_all.sh
> #!/bin/bash
>
> export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes
> export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes
>
> export BORG_CACHE_DIR=/backup/borg-cache
> export DATUM=$(date +"%d-%m-%Y_%H-%M")
> export ARCHIVENAME=archive-$DATUM
> export BORGBIN="/backup/bin/borg"
> export BASEPATH="/iscsi/lun1/borg-repos"
> export BASEPATHLOG="/iscsi/lun2/borg-logs"
> echo "hostname::archive-name orig-size compr-size dedup-size"
>
> for HOSTNAME in $(ls -1r /btrfspool/backup)
> do
> if [ ! -f /btrfspool/backup/$HOSTNAME/disabled ]
> then
> export REPOPATH="$BASEPATH/$HOSTNAME"
> export ARCHIVEPATH="$REPOPATH::$ARCHIVENAME"
> export LOG=$BASEPATHLOG/$HOSTNAME/borg_$DATUM.log
> export ERR=$BASEPATHLOG/$HOSTNAME/borg_$DATUM.err
>
> if [ ! -d $REPOPATH ]; then
> mkdir $REPOPATH
> mkdir $BASEPATHLOG/$HOSTNAME
> $BORGBIN init --encryption=none $REPOPATH
> fi
>
> cd /btrfspool/backup/$HOSTNAME/backup
> $BORGBIN create --filter=AME --info --list --stats --numeric-owner --compression lz4 $ARCHIVEPATH . >$LOG 2>$ERR
> echo $HOSTNAME::$ARCHIVENAME $($BORGBIN info $REPOPATH::$ARCHIVENAME |egrep "This archive"|cut -d ":" -f 2-)| /usr/bin/awk '{printf "%-60s %10s %2s %10s %2s %10s %2s\n",$1,$2,$3,$4,$5,$6,$7}'
> $BORGBIN prune --verbose --stats --keep-daily 14 --keep-weekly 8 --keep-monthly 6 $REPOPATH >>$LOG 2>>$ERR
> fi
> done
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
--
Adrian Klaver
adrian.klaver at aklaver.com
More information about the Borgbackup
mailing list