From ohnonot-github at posteo.de Wed Apr 27 00:40:56 2022 From: ohnonot-github at posteo.de (D.T.) Date: Wed, 27 Apr 2022 04:40:56 +0000 Subject: [Borgbackup] Prune too much? Message-ID: Hello, thanks for another great piece of FOSS! I have been using borg (1.2.0) succesfully for a few years now (on Archlinux, for this question). Seperate weekly repos for each partition, a script that iterates through them all. It prunes the archives like this: borg prune --stats --list -w 7 -m 7 -y 1 "$backup" Now there's one repository for a partition on a hard drive that is now missing, has been for a while, but it took me a while to catch up and remove it from the array of backup repositories on my script. I now noticed that of 15 archives in the repo, only 1 (the oldest) contains actual data, all others are empty. The oldest & only archive containing data is 1.5y old; I'm not 100% sure, but I think I still used the partition in question after that, and if I did I definitely added new files. I am slightly worried: could 'borg prune' ever result in a situation where I'm left with only empty archives? Or where it prunes the most recently added data? Thanks again, D.T. PS: The script sets the variables export BORG_UNKNOWN_UNENCRYPTED_REPO_ACCESS_IS_OK=yes export BORG_RELOCATED_REPO_ACCESS_IS_OK=yes I don't think it's relevant though... ___ Vaccines for everyone! Donate to the COVAX alliance: https://gogiveone.org/ (WHO) https://www.gavi.org/donate (Gavi) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: This is a digitally signed message part URL: From tw at waldmann-edv.de Wed Apr 27 05:43:32 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Apr 2022 11:43:32 +0200 Subject: [Borgbackup] Prune too much? In-Reply-To: References: Message-ID: <8998951a-0907-99da-ae0b-24e6c02cef39@waldmann-edv.de> > It prunes the archives like this: > borg prune --stats --list -w 7 -m 7 -y 1 "$backup" > > Now there's one repository for a partition on a hard drive that is now > missing, has been for a while, but it took me a while to catch up and > remove it from the array of backup repositories on my script. > > I now noticed that of 15 archives in the repo, only 1 (the oldest) > contains actual data, all others are empty. borg prune only looks at the archives' timestamps, it does not look at the archives' content (nor could it know whether the content is what you want it to be). > The oldest & only archive containing data is 1.5y old; I'm not 100% > sure, but I think I still used the partition in question after that, > and if I did I definitely added new files. I am not aware of bugs relating to the application of keep rules in borg prune. But sometimes is a bit difficult to see how it applied the rules and why the outcome is like it is. So there were users thinking it does not work correctly. But when analysing the reported cases, it always was found to work as designed / as documented. In borg 1.2, the output of borg prune was improved to also tell the rule that kept an archive to make it easier to see how it works and why it keeps a specific archive. Try --list. > I am slightly worried: could 'borg prune' ever result in a situation > where I'm left with only empty archives? Or where it prunes the most > recently added data? If the rules only keep archives that do not have the desired content, that is possible. It does not look at the content. Nor can it know what's the desired content. IIRC, we added some related error signalling code though: If you give a non-existing path to borg create, it will give an error msg and terminate with rc 2 (but it WILL create an archive containing all existing files other files, if any). If an include pattern never matched, it will give a warning msg and terminate with rc 1. General hints: - check borg output and return codes - maybe use borg create --list (maybe with or without --filter=AME) for better awareness about what will be in the created archive From tschoening at am-soft.de Mon May 2 04:44:01 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Mon, 2 May 2022 10:44:01 +0200 Subject: [Borgbackup] Does BORG take --file-cache into account when --read-special is used? Message-ID: <284648008.20220502104401@am-soft.de> Hi everyone, I've implemented a backup setup in which database dumps are executed remotely using SSH, data is forwarded using STDOUT and redirected by SSH into some created named pipe. Those pipes are read using "--read-special" and looking at the contents of the restored archives, things seem to work as expected. Just to be sure, those pipes always get re-created by deleting them after a backup and creating them before. Though, from my understanding in theory I could keep created ones and re-attach readers and writers as necessary. OTOH, those e.g. don't have a size, which is checked by default as part of the file-cache. So, how does BORG handle file-cache and special files like pipes? I would expect those to be ignored entirely? What's with other block devices, which might have e.g. a size always? Like in the following example: > $ borg create --read-special --chunker-params fixed,4194304 /path/to/repo::my-sdx /dev/sdX I guess you simply don't care and handle all of those non-file things the same, as it's easier and one can't know if a size is available at all in theory? Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From jdc at uwo.ca Sun May 8 20:08:26 2022 From: jdc at uwo.ca (Dan Christensen) Date: Mon, 9 May 2022 00:08:26 +0000 Subject: [Borgbackup] borg check errors after upgrade to 1.2.0 Message-ID: <87sfpjr2iu.fsf@uwo.ca> I upgraded from borg 1.1.7 to 1.2.0 on several different systems about a month ago. I use the fat binaries. This morning, my monthly "borg check" runs gave errors on three repositories on two systems. Note that I used the setup where several clients do their backups into the same repositories. System 1 runs Ubuntu 20.04. Two of the three repos on this machine now have errors: # borg check /Backups/borg/home.borg Index object count mismatch. committed index: 1166413 objects rebuilt index: 1166414 objects ID: 8a158ba7fdfae9b1373063a5bb5ea8ea6698c93ed7feff89ca6ff0a3c8842ebd rebuilt index: (18596, 199132336) committed index: Finished full repository check, errors found. # borg check /Backups/borg/system.borg Index object count mismatch. committed index: 2324200 objects rebuilt index: 2324202 objects ID: 1e20354918f4fdeb9cc0d677c28dffe1a383dd1b0db11ebcbc5ffb809d3c2b8a rebuilt index: (24666, 60168) committed index: ID: d9c516b5bf53f661a1a9d2ada08c8db7c33a331713f23e058cd6969982728157 rebuilt index: (3516, 138963001) committed index: Finished full repository check, errors found. The last monthly run was on April 10, just after I upgraded to 1.2.0, and there were no errors then. System 2 runs Debian buster. One of the three repos on this machine now has errors: # borg check /Backups/borg/system.borg Index object count mismatch. committed index: 2052187 objects rebuilt index: 2052188 objects ID: 6b734ed388e7e086af7107847c6b6d3d34a29c20e7e539ded71b32606cb857bd rebuilt index: (946, 15871355) committed index: Finished full repository check, errors found. On this system, I also did a clean borg check run on around April 10, using borg 1.2.0. I have used borg on these systems for years, and no hardware has changed recently. System 1 has the repos on a RAID 1 mdadm device with two SATA spinning disks. System 2 also has the repos on RAID 1 mdadm devices with two SATA disks, with lvm as a middle layer. In both cases, smartctl shows no issues for any of the drives, and memtester also shows no errors. Since the errors have happened on different machines within a month of upgrading to 1.2.0, I am concerned that this is a borg issue rather than a hardware issue. It is also suspicious to me that the error is the same in all cases, with a committed index not found. Hardware errors tend to produce garbage. I have not run repair yet. Is there anything I should do before running repair to try to figure out the issue? Each of the respositories is around 100GB, so I am reluctant to make copies. Thanks for any tips. I'm happy to file an issue if that would help. Dan From tw at waldmann-edv.de Mon May 9 07:30:49 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 9 May 2022 13:30:49 +0200 Subject: [Borgbackup] borg check errors after upgrade to 1.2.0 In-Reply-To: <87sfpjr2iu.fsf@uwo.ca> References: <87sfpjr2iu.fsf@uwo.ca> Message-ID: <323a655f-7c2d-3819-290d-d453c9e640a4@waldmann-edv.de> Hi Dan, that looks suspicious, please file a bug report (c&p this) and add some infos: does the defect also show in non-shared repos? are the repos strictly local? if not, give current borg version on server and client side. do you remmeber what version(s) you used before upgrading to 1.2.0? did you use --cleanup-commits (once or always)? did you already run borg compact? in the check error messages it gives "rebuilt index: (segment, offset)". can you do a ls -l repodir/data/SEGMENT_DIR/ for the segment directory containing that segment file (starts with same number)? (must be done before repair) can you do a borg debug get-obj CHUNKID and look what's in that chunk that was not in the repo index? (guess it'll work only after the repair) content? item metadata stream? archive metadata? ... > with a committed index not found. The index file is found, but it seems to miss one entry. > Each of the respositories is around 100GB, so I am reluctant to make > copies. Well, i'ld recommend working on a copy, if the repo is important. If you can't do a full copy you could try a hardlink-copy. Since 1.2.0 a hardlink copy should work (but not before), from the changelog: implement internal safe_unlink (was: truncate_and_unlink) function more safely: usually it does not truncate any more, only under "disk full" circumstances and only if there is only one hardlink. see: #6286 Cheers, Thomas From jdc at uwo.ca Mon May 9 08:54:19 2022 From: jdc at uwo.ca (Dan Christensen) Date: Mon, 9 May 2022 12:54:19 +0000 Subject: [Borgbackup] borg check errors after upgrade to 1.2.0 In-Reply-To: <323a655f-7c2d-3819-290d-d453c9e640a4@waldmann-edv.de> (Thomas Waldmann's message of "Mon, 9 May 2022 13:30:49 +0200") References: <87sfpjr2iu.fsf@uwo.ca> <323a655f-7c2d-3819-290d-d453c9e640a4@waldmann-edv.de> Message-ID: <87zgjqq32d.fsf@uwo.ca> Thanks, this is now issue #6687: https://github.com/borgbackup/borg/issues/6687 I added more information there. Note that the previous borg version I used was 1.1.17, not 1.1.7 as I accidentally wrote below. Dan On May 9, 2022, Thomas Waldmann wrote: > Hi Dan, > > that looks suspicious, please file a bug report (c&p this) and add some infos: > > does the defect also show in non-shared repos? > > are the repos strictly local? if not, give current borg version on server and client side. > > do you remmeber what version(s) you used before upgrading to 1.2.0? > > did you use --cleanup-commits (once or always)? > > did you already run borg compact? > > in the check error messages it gives "rebuilt index: (segment, offset)". > can you do a ls -l repodir/data/SEGMENT_DIR/ for the segment directory > containing that segment file (starts with same number)? (must be done > before repair) > > can you do a borg debug get-obj CHUNKID and look what's in that chunk > that was not in the repo index? (guess it'll work only after the > repair) > content? item metadata stream? archive metadata? ... > >> with a committed index not found. > > The index file is found, but it seems to miss one entry. > >> Each of the respositories is around 100GB, so I am reluctant to make >> copies. > > Well, i'ld recommend working on a copy, if the repo is important. > > If you can't do a full copy you could try a hardlink-copy. > > Since 1.2.0 a hardlink copy should work (but not before), from the changelog: > > implement internal safe_unlink (was: truncate_and_unlink) function > more safely: usually it does not truncate any more, only under "disk > full" > circumstances and only if there is only one hardlink. see: #6286 > > Cheers, Thomas > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From ndbecker2 at gmail.com Thu May 12 07:41:09 2022 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 12 May 2022 07:41:09 -0400 Subject: [Borgbackup] garbled message borg-1.2.0 Message-ID: Since updating to fedora36, which updates to borg-1.2.0, I'm seeing a garbled error message, which is a bit worrying. It is normal to see these messages about permission denied when backing up these two directories that are not readable by my user. But look closely. It should have said permission denied on the directory /home/nbecker/nbecker8/etc/openvpn/client But we see: Permission denied: 'client'.e=4096,include0,k=64,decim=1,freq=0.1.log.pickle The part that says: e=4096,include0,k=64,decim=1,freq=0.1.log.pickle is part of the name of an unrelated file. I hope this is just an artifact of the printing, and not pointing to memory corruption. Creating archive at "nbecker at 10.0.0.3:BACKUP/nbecker2::nbecker2-home-2022-05-12" /home/nbecker/nbecker8/etc/openvpn/client: dir_open: [Errno 13] Permission denied: 'client'.e=4096,include0,k=64,decim=1,freq=0.1.log.pickle /home/nbecker/nbecker8/etc/openvpn/server: dir_open: [Errno 13] Permission denied: 'server' /h -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Thu May 12 08:54:47 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 12 May 2022 14:54:47 +0200 Subject: [Borgbackup] garbled message borg-1.2.0 In-Reply-To: References: Message-ID: <60dcc1ed-2493-1103-44ad-fea8eb43168b@waldmann-edv.de> > permission?denied on the directory /home/nbecker/nbecker8/etc/openvpn/client > > But we see: > Permission denied: 'client'.e=4096,include0,k=64,decim=1,freq=0.1.log.pickle > > The part that says: > e=4096,include0,k=64,decim=1,freq=0.1.log.pickle > > is part of the name of an unrelated file.? I hope this is just an > artifact of the printing, and not pointing to memory corruption. That is strange. Can you write an as-simple-as-possible reproducer script and file a bug on github? From mason at ftlcomputing.com Wed May 18 13:05:52 2022 From: mason at ftlcomputing.com (Mason Schmitt) Date: Wed, 18 May 2022 10:05:52 -0700 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? Message-ID: Today I primarily use Borg to backup sparse KVM raw disk image files and soon I plan to use Borg to backup VM disks stored directly on ZFS zvols (in a Proxmox environment). My experience with backing up KVM sparse disk image files is that even if there have been very few changes to the VMs (mostly just writes to files in /var/log) backups still take several hours to a locally attached USB 3 disk. Presumably this is because the full file needs to be analyzed to see what has changed? I'm wondering whether the new feature in 1.2, "create --content-from-command", might be able to help reduce the time to create backups by feeding Borg with a list of changed blocks? Specifically, I'm wondering if zfs send could be used as an input to borg create --content-from-command? Or will the improvements in sparse file support and the chunker make a significant enough difference to backup times that I would be better off using Borg's native functionality and saving myself the complexity of the zfs send method? -- Mason -------------- next part -------------- An HTML attachment was scrubbed... URL: From lazyvirus at gmx.com Wed May 18 13:49:07 2022 From: lazyvirus at gmx.com (Bzzzz) Date: Wed, 18 May 2022 19:49:07 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: References: Message-ID: <20220518194907.0308f65e@msi.defcon1.lan> On Wed, 18 May 2022 10:05:52 -0700 Mason Schmitt via Borgbackup wrote: > Today I primarily use Borg to backup sparse KVM raw disk image files > and soon I plan to use Borg to backup VM disks stored directly on ZFS > zvols (in a Proxmox environment). > > My experience with backing up KVM sparse disk image files is that > even if there have been very few changes to the VMs (mostly just > writes to files in /var/log) backups still take several hours to a > locally attached USB 3 disk. I doubt it is USB3 fault, as a SSD on a Rpi4 read speed is an average of 235 MB/s. Did you change the --chunker-params or did you left it genuine ? Jean-Yves From tschoening at am-soft.de Wed May 18 13:32:16 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Wed, 18 May 2022 19:32:16 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: References: Message-ID: <1068198199.20220518193216@am-soft.de> Guten Tag Mason Schmitt via Borgbackup, am Mittwoch, 18. Mai 2022 um 19:05 schrieben Sie: > Today I primarily use Borg to backup sparse KVM raw disk image files and > soon I plan to use Borg to backup VM disks stored directly on ZFS zvols (in > a Proxmox environment). In my opinion it's better to configure ZFS as dir-storage in Proxmox and simply use files for VM-images instead of ZVOLs. Things like zfs-auto-snapshot, RSYNC, Borg, renaming files manually etc. work better files files. > My experience with backing up KVM sparse disk image files is that even if > there have been very few changes to the VMs (mostly just writes to files in > /var/log) backups still take several hours to a locally attached USB 3 > disk. Presumably this is because the full file needs to be analyzed to see > what has changed? Not sure what you mean with "analyzed", but the source file simply needs to be read entirely. Otherwise Borg wouldn't be able to know the changed blocks. You should look at your I/O closely, where exactly the bottleneck is, should be reading most likely with really only very few changes within VMs. > I'm wondering whether the new feature in 1.2, "create > --content-from-command", might be able to help reduce the time to create > backups by feeding Borg with a list of changed blocks? Specifically, I'm > wondering if zfs send could be used as an input to borg create > --content-from-command? ZFS sends data only meaningful for ZFS receive, Borg can't make too much sense of that. You would simply backup a DIFF of some data you can't easily restore in the end, you won't ever have a complete file. You might consider not using Borg at all, things heavily depend on how much you benefit from the built-in de-duplication. Remember that ZFS compresses as well already and even provides encryption, ZFS send+receive can be used for ZFS file systems on USB devices etc. as well and with tools like zfs-auto-snapshot you might get a history compareable to that of Borg. http://www.bolthole.com/solaris/zrep/ ZFS's de-duplication is pretty resource hungry, so Borg might perform better in that use-case. Using BorgMatic it might as well be easier to setup a backup process at all and it might be easier to backup different kinds of data. For example, I'm backing up different database dumps as well and Borg seems to do a pretty good job in de-duplicating those. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From mason at ftlcomputing.com Thu May 19 14:07:39 2022 From: mason at ftlcomputing.com (Mason Schmitt) Date: Thu, 19 May 2022 11:07:39 -0700 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: <20220518194907.0308f65e@msi.defcon1.lan> References: <20220518194907.0308f65e@msi.defcon1.lan> Message-ID: > > My experience with backing up KVM sparse disk image files is that > > even if there have been very few changes to the VMs (mostly just > > writes to files in /var/log) backups still take several hours to a > > locally attached USB 3 disk. > > I doubt it is USB3 fault, as a SSD on a Rpi4 read speed is an average of > 235 MB/s. > > Did you change the --chunker-params or did you left it genuine ? > I didn't change the chunker-params. Is there an optimum setting for backing up sparse, raw, VM images? Is the optimal setting different between 1.1.x and 1.2.x? I'm still running 1.1.17, which is the latest version on EPEL for CentOS 7. However, given that 1.2 has better sparse file support and an improved chunker, I'm wondering whether it might be worth stepping outside EPEL and instead using pip. Have you noticed a big performance improvement for large sparse files, like VM images, with 1.2? -- Mason -------------- next part -------------- An HTML attachment was scrubbed... URL: From mason at ftlcomputing.com Thu May 19 14:50:53 2022 From: mason at ftlcomputing.com (Mason Schmitt) Date: Thu, 19 May 2022 11:50:53 -0700 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: <1068198199.20220518193216@am-soft.de> References: <1068198199.20220518193216@am-soft.de> Message-ID: > > > Today I primarily use Borg to backup sparse KVM raw disk image files and > > soon I plan to use Borg to backup VM disks stored directly on ZFS zvols > (in > > a Proxmox environment). > > In my opinion it's better to configure ZFS as dir-storage in Proxmox > and simply use files for VM-images instead of ZVOLs. Things like > zfs-auto-snapshot, RSYNC, Borg, renaming files manually etc. work > better files files. > I'm on the fence when it comes to running VMs in files or directly on block devices. I have about 15 servers where I chose to use sparse, raw VM images, on XFS, on LVM thinpools, on MDADM. I'm considering building the next batch of servers using ZFS and debating whether to again use files or move to zvol backed VMs. Given that Borg can backup block devices, it seems like I could remove some complexity (files on a file system), while getting a slight performance improvement from eliminating the file system overhead. > My experience with backing up KVM sparse disk image files is that even if > > there have been very few changes to the VMs (mostly just writes to files > in > > /var/log) backups still take several hours to a locally attached USB 3 > > disk. Presumably this is because the full file needs to be analyzed to > see > > what has changed? > > Not sure what you mean with "analyzed", but the source file simply > needs to be read entirely. Otherwise Borg wouldn't be able to know the > changed blocks. You should look at your I/O closely, where exactly the > bottleneck is, should be reading most likely with really only very few > changes within VMs. > I might have misunderstood how Borg determines whether parts of a large file have changed. For a VM disk image, I thought that Borg had to break the whole file into chunks, compute a hash on each block and then look up the hashes in its index to determine if anything has changed. That's what I mean by "analyzed". For very large files, like a 1.5TB raw image for a file server VM, I assume that's a lot of work for the chunker. This is why I was looking at ZFS send. ZFS knows exactly which blocks have changed, so if very little has changed, Borg would have very little work to do. > I'm wondering whether the new feature in 1.2, "create > > --content-from-command", might be able to help reduce the time to create > > backups by feeding Borg with a list of changed blocks? Specifically, I'm > > wondering if zfs send could be used as an input to borg create > > --content-from-command? > > ZFS sends data only meaningful for ZFS receive, Borg can't make too > much sense of that. You would simply backup a DIFF of some data you > can't easily restore in the end, you won't ever have a complete file. > >From a purely conceptual viewpoint, I was thinking this might work something like this: - ZFS identifies changed blocks - ZFS sends the changed blocks to Borg (just the changed blocks, nothing else) - Borg chunks and hashes the input from ZFS - Borg compares the hashes against its index - Borg dedupes the input (probably shouldn't be any duplicates, because ZFS is only sending changes, unless there are multiple zvols stored in this repo) - Borg compresses and stores the chunks as a new backup With this simplistic conceptual view of how zfs send and borg might interact, my assumption is that each Borg backup would present a point in time view of the full zvol, not a bunch of diffs that would need to be replayed. Looking at this from an analogous perspective, if I were to use inotify to identify all changed files, I could feel those files to Borg and Borg would not have to do the work of reading through the file system. ZFS send would essentially be doing for blocks what inotify does for files. If this approach were feasible, it could drastically reduce the time required to back up very large block devices that have a low rate of change. It could also be adapted to LVM snapshots of thinvolumes (with the resurrection of something like lvmsync - https://github.com/mpalmer/lvmsync) > You might consider not using Borg at all, things heavily depend on how > much you benefit from the built-in de-duplication. Remember that ZFS > compresses as well already and even provides encryption, ZFS > send+receive can be used for ZFS file systems on USB devices etc. as > well and with tools like zfs-auto-snapshot you might get a history > compareable to that of Borg. > > http://www.bolthole.com/solaris/zrep/ > > ZFS's de-duplication is pretty resource hungry, so Borg might perform > better in that use-case. I have looked at using zfs send/receive, without Borg, one of the major shortcomings is dedupe. ZFS's dedupe is really resource intensive. Given I'm backing up full VM images, there is a lot of duplication of OS files and empty space (these are sparse images), which Borg 1.1.x appears to handle fairly well. ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 4.29 TB 233.05 GB 10.39 GB All archives: 1.07 PB 19.16 TB 1.86 TB Unique chunks Total chunks Chunk index: 1418451 153017084 ------------------------------------------------------------------------------ I love Borg's space efficiency and ability to run automated checks. The part I would love to improve upon is this (taken from the same log file that produced the summary above): *Duration: 5 hours 38 minutes 38.96 seconds* -- Mason -------------- next part -------------- An HTML attachment was scrubbed... URL: From lazyvirus at gmx.com Thu May 19 14:52:24 2022 From: lazyvirus at gmx.com (Bzzzz) Date: Thu, 19 May 2022 20:52:24 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: References: <20220518194907.0308f65e@msi.defcon1.lan> Message-ID: <20220519205224.1595a5a0@msi.defcon1.lan> On Thu, 19 May 2022 11:07:39 -0700 Mason Schmitt wrote: OOPS, sorry for the PM (who's the creep that changed MLs policy, which fails miserably each time there's more than one level in a thread !#@^) twice! :((( > > > My experience with backing up KVM sparse disk image files is that > > > even if there have been very few changes to the VMs (mostly just > > > writes to files in /var/log) backups still take several hours to a > > > locally attached USB 3 disk. > > > > I doubt it is USB3 fault, as a SSD on a Rpi4 read speed is an > > average of 235 MB/s. > > > > Did you change the --chunker-params or did you left it genuine ? > > > > I didn't change the chunker-params. Is there an optimum setting for > backing up sparse, raw, VM images? Is the optimal setting different > between 1.1.x and 1.2.x? Not that I know of, but people backupetting *<;-p) VMs usually lower minimal and maximal chunk sizes to get a maximum benefit of chunks deduplication, as a change in a large chunk means this chunk must be uploaded into the BB repo. See : https://borgbackup.readthedocs.io/en/stable/usage/notes.html#chunker-params https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html?highlight=chunker%20params https://borgbackup.readthedocs.io/en/stable/usage/create.html?highlight=chunker%20params Be careful, as changing chunk size(s) needs a brand new repo, which is logical, you can't compare chunks if their sizes are different. Also note (2nd link I think) that you can fix the chunk size AND that lowering chunk size requires more memory and ressources. Sooo, it'll take some time to find the good chunk sizes for your case - may be you should open another thread, asking people who backup VMs which values they are using. At home, I just have 2 VMs (virtualbox: w$7:50GB & w$XP:11GB) --chunker-params 16,23,16,4095 works better than the default (faster backups). In my notes, there's a pro guy who have multiple VMs at work using : --chunker-params 16,23,11,4095 which seems to suit quite well his needs. > I'm still running 1.1.17, which is the latest version on EPEL for > CentOS 7. I use the pip3 1.1.17 version on debians bulleye. > However, given that 1.2 has better sparse file support and > an improved chunker, I'm wondering whether it might be worth stepping > outside EPEL and instead using pip. Have you noticed a big > performance improvement for large sparse files, like VM images, with > 1.2? First rule of computing : never touch it when it's working. Second " " " : always let others deal with teething problems. So, I can't tell about 1.2 as I do not plan to flip to this version until the end of this year. Jean-Yves From tschoening at am-soft.de Thu May 19 15:02:41 2022 From: tschoening at am-soft.de (=?windows-1250?Q?Thorsten_Sch=F6ning?=) Date: Thu, 19 May 2022 21:02:41 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: <20220519205224.1595a5a0@msi.defcon1.lan> References: <20220518194907.0308f65e@msi.defcon1.lan> <20220519205224.1595a5a0@msi.defcon1.lan> Message-ID: <1873449718.20220519210241@am-soft.de> Guten Tag Bzzzz, am Donnerstag, 19. Mai 2022 um 20:52 schrieben Sie: > Sooo, it'll take some time to find the good chunk sizes for your case > - may be you should open another thread, asking people who backup VMs > which values they are using. I'm using Borg with default settings to backup multiple differently sized VMs, with different usage like file hosting, SVN, embedded databases, BTRFS snapshots etc. Overall size is around 950 GiB of data and backup size is between 2 and 3,5 hours mostly. Things heavily depend on changes, load on the backup target NAS etc. I'm reading from SSDs, forwarding thourhg unkonw LAN speed using SSH to some NAS-like box with not guaranteed speed. Doesn't sound that slow to me. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From lazyvirus at gmx.com Thu May 19 15:34:49 2022 From: lazyvirus at gmx.com (Bzzzz) Date: Thu, 19 May 2022 21:34:49 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: <1873449718.20220519210241@am-soft.de> References: <20220518194907.0308f65e@msi.defcon1.lan> <20220519205224.1595a5a0@msi.defcon1.lan> <1873449718.20220519210241@am-soft.de> Message-ID: <20220519213449.0a7e7f87@msi.defcon1.lan> On Thu, 19 May 2022 21:02:41 +0200 Thorsten Sch?ning wrote: > Guten Tag Bzzzz, > am Donnerstag, 19. Mai 2022 um 20:52 schrieben Sie: > > > Sooo, it'll take some time to find the good chunk sizes for your > > case > > - may be you should open another thread, asking people who backup > > VMs which values they are using. > > I'm using Borg with default settings to backup multiple differently > sized VMs, with different usage like file hosting, SVN, embedded > databases, BTRFS snapshots etc. Overall size is around 950 GiB of data > and backup size is between 2 and 3,5 hours mostly. Things heavily Not bad, not bad at all :) > depend on changes, load on the backup target NAS etc. I'm reading from > SSDs, forwarding thourhg unkonw LAN speed using SSH to some NAS-like > box with not guaranteed speed. > > Doesn't sound that slow to me. > > Mit freundlichen Gr??en > > Thorsten Sch?ning Jean-Yves From tschoening at am-soft.de Thu May 19 15:37:29 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Thu, 19 May 2022 21:37:29 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: References: <1068198199.20220518193216@am-soft.de> Message-ID: <148188907.20220519213729@am-soft.de> Guten Tag Mason Schmitt, am Donnerstag, 19. Mai 2022 um 20:50 schrieben Sie: > Given that Borg can backup block devices, it seems like I could remove some > complexity (files on a file system), while getting a slight performance > improvement from eliminating the file system overhead. In my opinion files are easier and more flexible to handle than raw block devices or similar. Just think of names, directories to organize them in, encrypzted vs. unencrypted storage, different tools like RSYNC etc. > I might have misunderstood how Borg determines whether parts of a large > file have changed. For a VM disk image, I thought that Borg had to break > the whole file into chunks, compute a hash on each block and then look up > the hashes in its index to determine if anything has changed. That's what > I mean by "analyzed". Which is simply reading the whole file in the end or the hashes couldn't be computed. So you either need to wait for all the reads or for the hash computing or the hash comparing or ... The reads seem most likely to me. > From a purely conceptual viewpoint, I was thinking this might work > something like this: > - ZFS identifies changed blocks > - ZFS sends the changed blocks to Borg (just the changed blocks, nothing > else) > - Borg chunks and hashes the input from ZFS > - Borg compares the hashes against its index > - Borg dedupes the input (probably shouldn't be any duplicates, because ZFS > is only sending changes, unless there are multiple zvols stored in this > repo) > - Borg compresses and stores the chunks as a new backup This makes Borg backup DIFFs produced by ZFS send and only being meaningful to ZFS receive. How do you think your restore process looks like? :-) It's restoring from Borg, sending the restored DIFFs to ZFS receive and hoping that you've got everything correct. Which is pretty unlikely, because ZFS send+receive heavily depend e.g. on correctly maintained snapshots to integrate the DIFF data into. I wouldn't do that... > With this simplistic conceptual view of how zfs send and borg might > interact, my assumption is that each Borg backup would present a point in > time view of the full zvol, not a bunch of diffs that would need to be > replayed. No, you have one full ZVOL, again in the streaming format of ZFS send only, and lots of individual ZFS send-DIFFs. Borg can't magically create a ZVOL of the given DIFFed streaming format. > Looking at this from an analogous perspective, if I were to use inotify to > identify all changed files, I could feel those files to Borg and Borg would > not have to do the work of reading through the file system. ZFS send would > essentially be doing for blocks what inotify does for files. Again, what exactly is your bottleneck? Find that first and try to improve afterwards. Remember that a backup is some frozen state at some point in time. How does inotify help you with that? It doesn't, either you need to feed each and every change into Borg, which is mostly not how backups are designed, or you need to maintain changes of files on your own and feed these individually into Borg. Which simply results in creating your own backup software with Borg being some storage layer only. Borg easily and fast detects if the VM-images have changed at all by looking at CTIME and stuff, inotify won't be of any help here. Don't do that. If backing up entire VM-images is too slow for any reason, e.g. do it less often, only some of those on different days and/or instead backup VM-contents pretty often. There are multiple ways to do so, I currently prefer mounting things using SSHFS. This allows Borg access to individual files, somewhat fast check their mod time+size and decide if those need to be backed up or not. There are other, even faster ways of course: Install Borg inside the VM and stuff like that. Windows-clients might become a problem, though, even with those you could export using SMB or alike. > If this approach were feasible, it could drastically reduce the time > required to back up very large block devices that have a low rate of > change. It could also be adapted to LVM snapshots of thinvolumes (with the > resurrection of something like lvmsync - > https://github.com/mpalmer/lvmsync) Or you normalize your servers to ZFS... > I have looked at using zfs send/receive, without Borg, one of the > major shortcomings is dedupe. ZFS's dedupe is really resource intensive. > Given I'm backing up full VM images, there is a lot of duplication of OS > files and empty space (these are sparse images), which Borg 1.1.x appears > to handle fairly well. Keep in mind that sparse blocks will kept sparse with send+receive and with such an approach really only changes themself will be sent+received+stored and that REALLY fast. In combination with possibly aggressive compression, you might not miss de-duplication too much. Though, Borg has some benefits of course, like dedup, being somewhat file system agnostic, working over SSH easily, BorgMatic as frontend... https://torsion.org/borgmatic/ > *Duration: 5 hours 38 minutes 38.96 seconds* Doesn't read that bad for your amount of data, doesn't it? Do those times include any repo checks as well already? Remember that you might run those less often as well. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From heiko.helmle at horiba.com Fri May 20 01:48:25 2022 From: heiko.helmle at horiba.com (Heiko Helmle) Date: Fri, 20 May 2022 05:48:25 +0000 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: <1873449718.20220519210241@am-soft.de> References: <20220518194907.0308f65e@msi.defcon1.lan> <20220519205224.1595a5a0@msi.defcon1.lan> <1873449718.20220519210241@am-soft.de> Message-ID: > -----Original Message----- > > Sooo, it'll take some time to find the good chunk sizes for your case > > - may be you should open another thread, asking people who backup VMs > > which values they are using. > > I'm using Borg with default settings to backup multiple differently sized VMs, > with different usage like file hosting, SVN, embedded databases, BTRFS > snapshots etc. Overall size is around 950 GiB of data and backup size is > between 2 and 3,5 hours mostly. Things heavily depend on changes, load on > the backup target NAS etc. I'm reading from SSDs, forwarding thourhg > unkonw LAN speed using SSH to some NAS-like box with not guaranteed > speed. Mind that it is very dependent on the format of the VM. I've found that the ProxMox backup format (VMA) (uncompressed) has nearly zero deduplication rate with default chunker settings, even when backing up the same VM twice without having it powered on in the meantime. But buzhash,16,23,16,4095 results in extremely good deduplication for the same workload. It's still not fast though but I'm not sure if different chunker settings would change speed much - I always assume the biggest tradeoff is memory. Same experience for mongodb dumpfiles. Default chunker settings of mongodump's standard "multiple files" format result in almost no dedup, even if run in quick succession, while 16.23.16.4095 gives surprisingly good results. But mind - in that situation too, memory consumption was not a primary concern. Cheers Heiko Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Gesch?ftsf?hrer: Ralph Lauxmann, Ergin Cansiz, Hiroyuki Urabe, Markus Bode, Dr. Robert Plank, Takashi Nagano, Dr. Hiroshi Nakamura, Tetsuhiro Habe. From tschoening at am-soft.de Fri May 20 03:05:35 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Fri, 20 May 2022 09:05:35 +0200 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: References: <20220518194907.0308f65e@msi.defcon1.lan> <20220519205224.1595a5a0@msi.defcon1.lan> <1873449718.20220519210241@am-soft.de> Message-ID: <33371684.20220520090535@am-soft.de> Guten Tag Heiko Helmle, am Freitag, 20. Mai 2022 um 07:48 schrieben Sie: > Mind that it is very dependent on the format of the VM. I've found > that the ProxMox backup format (VMA) (uncompressed) has nearly zero > deduplication rate with default chunker settings[...] I'm directly backing up running VMs in QCOW2 format after creating ZFS snapshots. 950 GiB of those with 7 days history results in 1,5 TiB accumulated archives, which is good enough for me currently to not need to think about customized chunker settings. That should be some worst case already in theory because most VMs use BTRFS/ZFS with default compression already. I guess that makes dedup a lot more difficult. > Same experience for mongodb dumpfiles. Default chunker settings of > mongodump's standard "multiple files" format result in almost no > dedup, even if run in quick succession, while 16.23.16.4095 gives > surprisingly good results. But mind - in that situation too, memory > consumption was not a primary concern. The most important thing seems to be to get a somewhat stable, uncompressed dump output in the first place. I have some Postgres database storing large binary files as large objects and Borg seems to handle that pretty well using the custom, but uncompressed dump format. It's only very few GiBs of changes per day for an otherwise ~200 GiB dump, again with default chunker settings. Depending on the restore needs and things like temporary storage and stuff, other formats like plain text or dumping into a temporary archive using directory formats might make sense as well. Am using plain text for small MySQL databases, seems to work pretty well, too. Though, am trying to avoid additional copies into the file system by piping dumps directly into named pipes read by Borg, so uncompressed, one-file-based dumps are my favourite currently. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From heiko.helmle at horiba.com Fri May 20 04:46:38 2022 From: heiko.helmle at horiba.com (Heiko Helmle) Date: Fri, 20 May 2022 08:46:38 +0000 Subject: [Borgbackup] Reducing backup times for raw VM images, with zfs send? In-Reply-To: <33371684.20220520090535@am-soft.de> References: <20220518194907.0308f65e@msi.defcon1.lan> <20220519205224.1595a5a0@msi.defcon1.lan> <1873449718.20220519210241@am-soft.de> <33371684.20220520090535@am-soft.de> Message-ID: > Guten Tag Heiko Helmle, > am Freitag, 20. Mai 2022 um 07:48 schrieben Sie: > > > Mind that it is very dependent on the format of the VM. I've found > > that the ProxMox backup format (VMA) (uncompressed) has nearly zero > > deduplication rate with default chunker settings[...] > > I'm directly backing up running VMs in QCOW2 format after creating ZFS > snapshots. 950 GiB of those with 7 days history results in 1,5 TiB accumulated > archives, which is good enough for me currently to not need to think about > customized chunker settings. I've had good results with QCOW2 and VMDK, but not with VMA - and unfortunately, that's what Proxmox gives me in our configuration. > The most important thing seems to be to get a somewhat stable, > uncompressed dump output in the first place. I have some Postgres > database storing large binary files as large objects and Borg seems to handle > that pretty well using the custom, but uncompressed dump format. It's only > very few GiBs of changes per day for an otherwise > ~200 GiB dump, again with default chunker settings. Same with MySQL. As the dump there is plain SQL text and usually always sorted the same, dumps tend to perform well in dedeup. I have the feeling that both Proxmox VMA and mongodump have highly varying block headers that are too close together for the default minimum chunk size of borg, resulting in lots of unique chunks. For mongodb the obvious solution is not using mongodump at all and instead taking the original files from a volume snapshot. This has lots of advantages with the slight disadvantage that it's less portable and harder to set up. Cheers Heiko Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Gesch?ftsf?hrer: Ralph Lauxmann, Ergin Cansiz, Hiroyuki Urabe, Markus Bode, Dr. Robert Plank, Takashi Nagano, Dr. Hiroshi Nakamura, Tetsuhiro Habe. From ohnonot-github at posteo.de Sat May 21 10:19:06 2022 From: ohnonot-github at posteo.de (D.T.) Date: Sat, 21 May 2022 14:19:06 +0000 Subject: [Borgbackup] Prune too much? In-Reply-To: <8998951a-0907-99da-ae0b-24e6c02cef39@waldmann-edv.de> References: <8998951a-0907-99da-ae0b-24e6c02cef39@waldmann-edv.de> Message-ID: A belated thanks for your quick reply. I have now modified my backup script to skip the pruning if borg create returns anything other than 0. On Wed, 2022-04-27 at 11:43 +0200, Thomas Waldmann wrote: > > It prunes the archives like this: > > borg prune --stats --list -w 7 -m 7 -y 1 "$backup" > > > > Now there's one repository for a partition on a hard drive that is > > now > > missing, has been for a while, but it took me a while to catch up > > and > > remove it from the array of backup repositories on my script. > > > > I now noticed that of 15 archives in the repo, only 1 (the oldest) > > contains actual data, all others are empty. > > borg prune only looks at the archives' timestamps, it does not look > at > the archives' content (nor could it know whether the content is what > you > want it to be). > > > The oldest & only archive containing data is 1.5y old; I'm not 100% > > sure, but I think I still used the partition in question after > > that, > > and if I did I definitely added new files. > > I am not aware of bugs relating to the application of keep rules in > borg > prune. > > But sometimes is a bit difficult to see how it applied the rules and > why > the outcome is like it is. So there were users thinking it does not > work > correctly. But when analysing the reported cases, it always was found > to > work as designed / as documented. > > In borg 1.2, the output of borg prune was improved to also tell the > rule > that kept an archive to make it easier to see how it works and why it > keeps a specific archive. Try --list. > > > I am slightly worried: could 'borg prune' ever result in a > > situation > > where I'm left with only empty archives? Or where it prunes the > > most > > recently added data? > > If the rules only keep archives that do not have the desired content, > that is possible. It does not look at the content. Nor can it know > what's the desired content. > > > IIRC, we added some related error signalling code though: > > If you give a non-existing path to borg create, it will give an error > msg and terminate with rc 2 (but it WILL create an archive containing > all existing files other files, if any). > > If an include pattern never matched, it will give a warning msg and > terminate with rc 1. > > > General hints: > > - check borg output and return codes > > - maybe use borg create --list (maybe with or without --filter=AME) > for > better awareness about what will be in the created archive > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup ___ Vaccines for everyone! Donate to the COVAX alliance: https://gogiveone.org/?(WHO) https://www.gavi.org/donate?(Gavi) ___ Vaccines for everyone! Donate to the COVAX alliance: https://gogiveone.org/ (WHO) https://www.gavi.org/donate (Gavi) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 659 bytes Desc: This is a digitally signed message part URL: From tw at waldmann-edv.de Fri May 27 09:33:43 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 27 May 2022 15:33:43 +0200 Subject: [Borgbackup] borg 1.2 pattern matching changes (compared to < 1.2) Message-ID: long ago, we changed the pattern matching to be better aligned with how borg archives stuff: borg always archives relative paths. e.g. filesystem /home/user/.bashrc will become home/user/.bashrc in the archive. thus, we changed the pattern processing to apply to the paths you also see when you list a borg archive later. no leading slashes! i am sorry that i overlooked this and forgot to include a prominent note about this change into the 1.2.0 releases' compatibility notes, we added this now (and also fixed the docs which where also not updated everywhere yet): you can see the docs / examples changes in this diff: https://github.com/borgbackup/borg/pull/6718/files please understand that borg can not automatically deal with leading slashes in patterns (except in the most primitive cases), so you need to adapt them manually. just do not try to match the leading slash, it is not there. updated docs / changelog will be releases together with other fixes in 1.2.1 later. From tschoening at am-soft.de Sun May 29 17:26:36 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Sun, 29 May 2022 23:26:36 +0200 Subject: [Borgbackup] When to use "borg upgrade" after upgrading 1.1.x to 1.2.0? Message-ID: <412607107.20220529232636@am-soft.de> Hi everyone, the release notes of upgrading Borg 1.1.16 to 1.1.17 explictily mention that "borg upgrade" is NOT necessary. The release notes of Borg 1.2.0 do not seem to mention "borg upgrade" at all. The docs of "borg upgrade" mention when it's NOT encessary as well, with some special exceptions. OTOH, the release notes of 1.2.0 explicitly mention new archive metadata and performance improvements for some operations: > info: use a pre12-meta cache to accelerate stats for borg < 1.2 > archives. the first time borg info is invoked on a borg 1.1 repo, it > can take a rather long time computing and caching some stats values > for 1.1 archives, which borg 1.2 archives have in their archive > metadata structure.[...] https://borgbackup.readthedocs.io/en/stable/changes.html#change-log It's unclear to me when "borg upgrade" is actually needed or recommended to e.g. get best performance of operations like "info". Are 1.2 archive formats written always simply by using a 1.2 binary to create/write the archives,s o upgrade is not necessary? Would upgrade rewrite all archives with 1.2 at all? The docs aren't too clear what "upgrade" actually does. Reads like creating new repos with current state and data however the used binary does so. What does "local Borg repository" mean anyway? Does it still work e.g. with a client/server setup? In e.g. my case I don't have access to exec the server side process manually locally on the box where it runs. It's always remote. Thanks! Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From imperator at jedimail.de Mon May 30 01:38:53 2022 From: imperator at jedimail.de (Sascha Ternes) Date: Mon, 30 May 2022 07:38:53 +0200 Subject: [Borgbackup] When to use "borg upgrade" after upgrading 1.1.x to 1.2.0? In-Reply-To: <412607107.20220529232636@am-soft.de> References: <412607107.20220529232636@am-soft.de> Message-ID: Hello Thorsten, Am 29.05.22 um 23:26 schrieb Thorsten Sch?ning: > the release notes of upgrading Borg 1.1.16 to 1.1.17 explictily > mention that "borg upgrade" is NOT necessary. The release notes of > Borg 1.2.0 do not seem to mention "borg upgrade" at all. The docs of > "borg upgrade" mention when it's NOT encessary as well, with some > special exceptions. 'borg upgrade' is not needed when upgrading from 1.1.x to 1.2.0. Just do the checklist in the 1.2.0 change log. 'borg compact --cleanup-commits' is the most important part here. You might notice that the first 'borg info'of 1.2.0 will take much longer than normal, because there are changes in the cache that need to be rebuilt. In short: - 1.1.17: 'borg check' - 1.2.0: 'borg compact --cleanup-commits' - 1.2.0: 'borg info /repo::' You should add the new 'borg compact' command to your pruning process, because 'borg prune' does not free the data anymore, it just removes the archives from the repository structure. Sascha From tschoening at am-soft.de Mon May 30 10:37:08 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Mon, 30 May 2022 16:37:08 +0200 Subject: [Borgbackup] When to use "borg upgrade" after upgrading 1.1.x to 1.2.0? In-Reply-To: References: <412607107.20220529232636@am-soft.de> Message-ID: <617317694.20220530163708@am-soft.de> Guten Tag Sascha Ternes, am Montag, 30. Mai 2022 um 07:38 schrieben Sie: > 'borg upgrade' is not needed when upgrading from 1.1.x to 1.2.0. But when is it needed or at least recommended and what does it do in the end? :-) E.g. I asked myself about why running possibly long-running "borg info", like advised in the release noters, instead of simply upgrading the whole repo. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From imperator at jedimail.de Mon May 30 12:11:14 2022 From: imperator at jedimail.de (Sascha Ternes) Date: Mon, 30 May 2022 18:11:14 +0200 Subject: [Borgbackup] When to use "borg upgrade" after upgrading 1.1.x to 1.2.0? In-Reply-To: <617317694.20220530163708@am-soft.de> References: <412607107.20220529232636@am-soft.de> <617317694.20220530163708@am-soft.de> Message-ID: Hello Thorsten, Am 30.05.22 um 16:37 schrieb Thorsten Sch?ning: > But when is it needed or at least recommended and what does it do in > the end? :-) see https://borgbackup.readthedocs.io/en/stable/usage/upgrade.html: - Upgrade from attic to borg - Upgrade from 1.0.8 to a higher version (see also https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability-cve-2016-10099) Sascha From tschoening at am-soft.de Mon May 30 12:24:27 2022 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Mon, 30 May 2022 18:24:27 +0200 Subject: [Borgbackup] When to use "borg upgrade" after upgrading 1.1.x to 1.2.0? In-Reply-To: References: <412607107.20220529232636@am-soft.de> <617317694.20220530163708@am-soft.de> Message-ID: <1062886139.20220530182427@am-soft.de> Guten Tag Sascha Ternes, am Montag, 30. Mai 2022 um 18:11 schrieben Sie: > see https://borgbackup.readthedocs.io/en/stable/usage/upgrade.html: I did already and it doesn't answer my questions. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: 05151- 9468- 0 Tel: 05151- 9468-55 Fax: 05151- 9468-88 Mobil: 0178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 (0)515 94 68 - 0 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From clickwir at gmail.com Mon May 30 23:47:04 2022 From: clickwir at gmail.com (Zack Coffey) Date: Mon, 30 May 2022 21:47:04 -0600 Subject: [Borgbackup] When to use "borg upgrade" after upgrading 1.1.x to 1.2.0? In-Reply-To: <1062886139.20220530182427@am-soft.de> References: <412607107.20220529232636@am-soft.de> <617317694.20220530163708@am-soft.de> <1062886139.20220530182427@am-soft.de> Message-ID: In the context of the original subject: "When to use "borg upgrade" after upgrading 1.1.x to 1.2.0?" The answer is there. "You do not need to run it when" "except when noted otherwise in the changelog". So, it's pretty simple. It wasn't listed in the changelog, you aren't upgrading from attic or borg 1.0.8, so you don't need it. On Mon, May 30, 2022 at 10:24 AM Thorsten Sch?ning wrote: > Guten Tag Sascha Ternes, > am Montag, 30. Mai 2022 um 18:11 schrieben Sie: > > > see https://borgbackup.readthedocs.io/en/stable/usage/upgrade.html: > > I did already and it doesn't answer my questions. > > Mit freundlichen Gr??en > > Thorsten Sch?ning > > -- > AM-SoFT IT-Service - Bitstore Hameln GmbH > Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK > > E-Mail: Thorsten.Schoening at AM-SoFT.de > Web: http://www.AM-SoFT.de/ > > Tel: 05151- 9468- 0 > Tel: 05151- 9468-55 > Fax: 05151- 9468-88 > Mobil: 0178-8 9468-04 > > AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 > Hameln > AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska > > > F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. > > Mit freundlichen Gr??en, > > Thorsten Sch?ning > > > Telefon: +49 (0)515 94 68 - 0 > Fax: > E-Mail: TSchoening at am-soft.de > > AM-Soft IT-Service - Bitstore Hameln GmbH > Brandenburger Stra?e 7c > 31789 Hameln > > Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte > Informationen und ist ausschliesslich f?r den Adressaten bestimmt. > Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten > ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail > irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und > vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail > bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung > oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im > Vertrauen auf erlangte Information untersagt. > > This e-mail may contain confidential and/or privileged information and is > intended solely for the addressee. Access to this email by anyone else is > unauthorized. If you are not the intended recipient (or have received this > e-mail in error) please notify the sender immediately and destroy this > e-mail. If you are not the intended recipient, any disclosure, copying, > distribution or any action taken or omitted to be taken in reliance on it, > is prohibited and may be unlawful. > > Hinweise zum Datenschutz: bitstore.group/datenschutz > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Sun Jun 5 10:55:06 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 5 Jun 2022 16:55:06 +0200 Subject: [Borgbackup] borgbackup 1.1.18 released Message-ID: <7a28a11b-2dfb-2f22-31f2-c6c0111a04f7@waldmann-edv.de> Just released borgbackup 1.1.18 (likely the last 1.1.x release). Details see there: https://github.com/borgbackup/borg/releases/tag/1.1.18 From tw at waldmann-edv.de Sun Jun 5 17:55:51 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 5 Jun 2022 23:55:51 +0200 Subject: [Borgbackup] borgbackup 1.2.1 released! Message-ID: <704b92a9-5cd4-4cb7-966e-f4e96f33d400@waldmann-edv.de> borgbackup 1.2.1 was just released. Details see there: https://github.com/borgbackup/borg/releases/tag/1.2.1 From tw at waldmann-edv.de Sun Jun 26 10:58:09 2022 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 26 Jun 2022 16:58:09 +0200 Subject: [Borgbackup] borgbackup 2.0.0a2 released - only for testing! Message-ID: See there: https://github.com/borgbackup/borg/releases/tag/2.0.0a2 Read the changelog! This is a BREAKING release. Play around with it, find bugs, try to convert borg 1.1/1.2 repos, give feedback in the discussion issue or via the issue tracker. CLI interface is different, read the docs, adapt scripts and wrappers. Be very careful, never directly work on production repos with this. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393