From mszpak at wp.pl Mon Apr 2 20:12:33 2018 From: mszpak at wp.pl (=?UTF-8?Q?Marcin_Zaj=c4=85czkowski?=) Date: Tue, 3 Apr 2018 02:12:33 +0200 Subject: [Borgbackup] Determine which files greatly increased backup size Message-ID: Hi, I'm evaluating Borg Backup as a potential replacement of rdiff-backup. I really like its reduplication and compression, however having a daily backup of my Firefox profiles (around 300MB) rdiff-backup has an increment of ~35MB, while with Borg it is ~65MB. As it is a daily backup it give around 10GB more space allocated per year. Sure, with proper data retention it would be less, but I would like to investigate it and maybe play with the bucket size (or exclude some wonky files from a backup). Is it possible to get detailed information at the end of a backup creation how much given file increased a backup (archive) size? Marcin From tw at waldmann-edv.de Tue Apr 3 11:33:36 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 3 Apr 2018 17:33:36 +0200 Subject: [Borgbackup] Determine which files greatly increased backup size In-Reply-To: References: Message-ID: <1c9c2786-d22b-45e4-5b89-20b0e99f577a@waldmann-edv.de> > I'm evaluating Borg Backup as a potential replacement of rdiff-backup. I > really like its reduplication and compression, however having a daily > backup of my Firefox profiles (around 300MB) rdiff-backup has an > increment of ~35MB, while with Borg it is ~65MB. As it is a daily backup > it give around 10GB more space allocated per year. > > Sure, with proper data retention it would be less, but I would like to > investigate it and maybe play with the bucket size (or exclude some > wonky files from a backup). You can play with --chunker-params, but be warned that having smaller target chunk size also means having more chunks to manage, requiring more time and space (on disk, in memory) to deal with them. The latter is the reason why borg targets 2MiB chunks by default. > Is it possible to get detailed information at the end of a backup > creation how much given file increased a backup (archive) size? No. You can use --stats and use some special placeholders for borg list, but I guess that won't help enough. What usually helps a bit is to keep the --stats and --list output of a borg create run and do educated guesses based on these. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From ndbecker2 at gmail.com Fri Apr 6 14:10:32 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 06 Apr 2018 18:10:32 +0000 Subject: [Borgbackup] msgpack requirements? Message-ID: pip3 check borgbackup 1.1.5 has requirement msgpack-python<0.5.0,>=0.4.6, but you have msgpack-python 0.5.6. Should I be worried? -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Fri Apr 6 14:47:21 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 6 Apr 2018 20:47:21 +0200 Subject: [Borgbackup] msgpack requirements? In-Reply-To: References: Message-ID: On 06.04.2018 20:10, Neal Becker wrote: > pip3 check > borgbackup 1.1.5 has requirement msgpack-python<0.5.0,>=0.4.6, but you > have msgpack-python 0.5.6. See there: https://github.com/borgbackup/borg/issues/3753 > Should I be worried? Maybe not, but which of the 0.5.x versions has which (sometimes minor / cosmetic) problems has still to be documented in that ticket. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From tw at waldmann-edv.de Sun Apr 8 14:20:27 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 8 Apr 2018 20:20:27 +0200 Subject: [Borgbackup] borgbackup 1.0.12 released! Message-ID: <9a80dd30-cf3a-1c04-6163-1ff7c26722c0@waldmann-edv.de> Released borgbackup 1.0.12 with misc. bug fixes. https://github.com/borgbackup/borg/releases/tag/1.0.12 This is for users who still want to stay on 1.0.x. Everybody else should rather use 1.1.5. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mailinglists at xgm.de Sun Apr 8 14:47:47 2018 From: mailinglists at xgm.de (Florian Lindner) Date: Sun, 08 Apr 2018 20:47:47 +0200 Subject: [Borgbackup] borg check stalls Message-ID: <7867100.a5rP4No7UP@horus> Hello, borg list or info, as well as the usual backups work just fine, e.g. % borg --debug list using builtin fallback logging configuration 35 self tests completed in 0.07 seconds SSH command line: ['ssh', '-p', '23', 'u123 at u123.your-storagebox.de', 'borg', 'serve', '--umask=077', '--debug'] TAM-verified manifest security: read previous location 'ssh://u123 at u123.your-storagebox.de:23/./Backup' security: read manifest timestamp '2018-04-08T16:08:29.849300' security: determined newest manifest timestamp as 2018-04-08T16:08:29.849300 security: repository checks ok, allowing access horus-2018-02-25T00:00:58 Sun, 2018-02-25 00:01:01 [83890f623fd4d7e7bf4a6921564bead27bf1f0efd8a9f749a20c2f24c600901b] [...] but borg check stops at the SSH connection: % borg --debug check using builtin fallback logging configuration 35 self tests completed in 0.07 seconds SSH command line: ['ssh', '-p', '23', 'u123 at u123.your-storagebox.de', 'borg', 'serve', '--umask=077', '--debug'] After about 10 minutes with no output, I use Ctrl-C to cancel: ^CRemoteRepository: 148 B bytes sent, 66 B bytes received, 3 messages sent Connection closed by remote host Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 152, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 306, in do_check if not repository.check(repair=args.repair, save_space=args.save_space): File "/usr/lib/python3.6/site-packages/borg/remote.py", line 476, in do_rpc return self.call(f.__name__, named, **extra) File "/usr/lib/python3.6/site-packages/borg/remote.py", line 707, in call for resp in self.call_many(cmd, [args], **kw): File "/usr/lib/python3.6/site-packages/borg/remote.py", line 800, in call_many r, w, x = select.select(self.r_fds, w_fds, self.x_fds, 1) File "/usr/lib/python3.6/site-packages/borg/helpers.py", line 2234, in handler raise exc_cls KeyboardInterrupt During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 4253, in main exit_code = archiver.run(args) File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 4185, in run return set_ec(func(args)) File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 152, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3.6/site-packages/borg/remote.py", line 630, in __exit__ self.rollback() File "/usr/lib/python3.6/site-packages/borg/remote.py", line 476, in do_rpc return self.call(f.__name__, named, **extra) File "/usr/lib/python3.6/site-packages/borg/remote.py", line 707, in call for resp in self.call_many(cmd, [args], **kw): File "/usr/lib/python3.6/site-packages/borg/remote.py", line 807, in call_many raise ConnectionClosed() borg.remote.ConnectionClosed: Connection closed by remote host Platform: Linux horus 4.15.14-1-ARCH #1 SMP PREEMPT Wed Mar 28 17:34:29 UTC 2018 x86_64 Linux: arch Borg: 1.1.5 Python: CPython 3.6.4 PID: 11786 CWD: /home/florian sys.argv: ['/bin/borg', '--debug', 'check'] SSH_ORIGINAL_COMMAND: None Using borg check on the same server (different path albeit) works fine, with borg 1.0.9 from Debian stable. Any idea what is going wrong with borg check here? Thanks, Florian From tw at waldmann-edv.de Mon Apr 9 09:26:34 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 9 Apr 2018 15:26:34 +0200 Subject: [Borgbackup] borg check stalls In-Reply-To: <7867100.a5rP4No7UP@horus> References: <7867100.a5rP4No7UP@horus> Message-ID: <7546a47f-e51a-e8de-09cb-992ae00a4c58@waldmann-edv.de> Hi Florian, I copied this to the github issue tracker: https://github.com/borgbackup/borg/issues/3763 If you have a github account, let's continue there. Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From liori at exroot.org Sat Apr 14 17:47:42 2018 From: liori at exroot.org (Tomasz Melcer) Date: Sat, 14 Apr 2018 23:47:42 +0200 Subject: [Borgbackup] Edge cases in disk image example in documentation Message-ID: <3ade6efc-2c7a-163d-b4fe-040b8587f8dc@exroot.org> Hello, I started looking at the suggested use of borg to back up disk images, [1]. [1] https://borgbackup.readthedocs.io/en/stable/deployment/image-backup.html I've noticed that the scripts shown there may have problems performing backups correctly in some edge cases. Firstly, the suggested scripts don't deal with extended partitions. Only the main partition table is backed up. This may result in loss of data or difficulties restoring the backups. Secondly, the size of the header is defined as everything before the "first" partition. However, the script considers the first partition as the one listed as first in the partition table, and not the one that is starting from the lowest sector. Curiously, the first partition in the partition table doesn't have to be the one being "physically" the first one on the disk. If this (rare) condition happens, the `hostname-partinfo` archive will be needlessly big. The way suggested by the `sfdisk` man page to deal with these problems is to use its `--backup` switch ? a bit bothersome to work with, but it seems to work for me. I have a script that performs a similar procedure and which has been tested in these two corner cases. However, instead of storing partitions as separate archives, it firstly builds a "clean" version of the disk image, with zeroed-out sectors not used by filesystems or sectors between partitions, and only then stores the clean image as an archive. I found this procedure to be a bit more useful for my use case ? I noticed I can start a virtual machine and choose a disk image straight from `borg mount`, which is pretty amazing! I've posted my script at [2]. As it is different, it wouldn't outright replace the scripts from the page linked above. If you think it's proper to add it to the documentation next to the existing script, I'm willing to submit a patch. [2] https://gist.github.com/liori/ceac2a0c2fc7de1775a2585c4209f51b -- Tomasz Melcer From geek at airmail.cc Mon Apr 23 11:58:11 2018 From: geek at airmail.cc (geek at airmail.cc) Date: Mon, 23 Apr 2018 17:58:11 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? Message-ID: <8461b062b98a657179e25bec62149142@airmail.cc> Hello, I have a borg backup of 394 GB. It seems that due to an administrative failure last 2252 chunks (files) out of 82214 are missing. I.e. file index.82214 was rebuilt (renamed) to index.79962 during borg -list execution. Since the manifest is exactly on that last files - it is missing. I am running now the following command: $ borg -p -v check --repair backup 'check --repair' is an experimental feature that might result in data loss. Type 'YES' if you understand this and want to continue: YES Starting repository check Starting repository index check Completed repository check, no problems found. <<< This took around 4 hours, progress percentage was reported. Starting archive consistency check... Enter passphrase for key backup: Repository manifest not found! Rebuilding missing manifest, this might take some time... It's rebuilding ~100 hours as of now. Progress is not reported. Can somebody have an idea how long will it take? I mean - more days, weeks, years? Is there any chance that it'll be finished in a reasonable time? As I told the repository check (without errors) was done in 4 hours and progress percentage was reported. Thank you! From tw at waldmann-edv.de Mon Apr 23 12:21:11 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 23 Apr 2018 18:21:11 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: <8461b062b98a657179e25bec62149142@airmail.cc> References: <8461b062b98a657179e25bec62149142@airmail.cc> Message-ID: > I have a borg backup of 394 GB. It seems that due to an administrative > failure last 2252 chunks (files) out of 82214 are missing. You mean segment files, not chunks. It's likely more chunks as multiple chunks are aggregated into one segment file. > I.e. file index.82214 was rebuilt (renamed) to index.79962 during borg > -list execution. Sounds like a transaction rollback. > Since the manifest is exactly on that last files - it is missing. > > I am running now the following command: > > $ borg -p -v check --repair backup > 'check --repair' is an experimental feature that might result in data loss. > Type 'YES' if you understand this and want to continue: YES > Starting repository check > Starting repository index check > Completed repository check, no problems found. <<< This took around 4 > hours, progress percentage was reported. > Starting archive consistency check... > Enter passphrase for key backup: > Repository manifest not found! > Rebuilding missing manifest, this might take some time... > > It's rebuilding ~100 hours as of now. Progress is not reported. Can > somebody have an idea how long will it take? This depends on a lot of factors. It will read (and authenticate, decrypt, decompress) ALL data (chunks) from the repo, looking for stuff that looks like an archive and rebuild the manifest with that information. > As I told the repository check (without errors) was done in 4 hours and > progress percentage was reported. It will also report progress when it starts checking each archive, but looks like the manifest rebuild does not show internal progress. Maybe file a bug, so we look into that missing piece. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From geek at airmail.cc Mon Apr 23 12:56:54 2018 From: geek at airmail.cc (geek at airmail.cc) Date: Mon, 23 Apr 2018 18:56:54 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: References: <8461b062b98a657179e25bec62149142@airmail.cc> Message-ID: >> I.e. file index.82214 was rebuilt (renamed) to index.79962 during borg >> -list execution. > > Sounds like a transaction rollback. What does it mean? How could borg roll the transaction back even without asking for the repokey? > This depends on a lot of factors. It will read (and authenticate, > decrypt, decompress) ALL data (chunks) from the repo, looking for > stuff that looks like an archive and rebuild the manifest with that > information. Thank you for the explanation. Is there any way to check that a) the process is not stuck b) to get any estimate where is it now and how long still to go? > It will also report progress when it starts checking each archive, but > looks like the manifest rebuild does not show internal progress. > > Maybe file a bug, so we look into that missing piece. Done, issue #3787. On 2018-04-23 18:21, Thomas Waldmann wrote: >> I have a borg backup of 394 GB. It seems that due to an administrative >> failure last 2252 chunks (files) out of 82214 are missing. > > You mean segment files, not chunks. > > It's likely more chunks as multiple chunks are aggregated into one > segment file. > >> I.e. file index.82214 was rebuilt (renamed) to index.79962 during borg >> -list execution. > > Sounds like a transaction rollback. > >> Since the manifest is exactly on that last files - it is missing. >> >> I am running now the following command: >> >> $ borg -p -v check --repair backup >> 'check --repair' is an experimental feature that might result in data >> loss. >> Type 'YES' if you understand this and want to continue: YES >> Starting repository check >> Starting repository index check >> Completed repository check, no problems found. <<< This took around 4 >> hours, progress percentage was reported. >> Starting archive consistency check... >> Enter passphrase for key backup: >> Repository manifest not found! >> Rebuilding missing manifest, this might take some time... >> >> It's rebuilding ~100 hours as of now. Progress is not reported. Can >> somebody have an idea how long will it take? > > This depends on a lot of factors. It will read (and authenticate, > decrypt, decompress) ALL data (chunks) from the repo, looking for > stuff that looks like an archive and rebuild the manifest with that > information. > >> As I told the repository check (without errors) was done in 4 hours >> and progress percentage was reported. > > It will also report progress when it starts checking each archive, but > looks like the manifest rebuild does not show internal progress. > > Maybe file a bug, so we look into that missing piece. From tw at waldmann-edv.de Mon Apr 23 13:07:41 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 23 Apr 2018 19:07:41 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: References: <8461b062b98a657179e25bec62149142@airmail.cc> Message-ID: <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> > What does it mean? How could borg roll the transaction back even without > asking for the repokey? Repo-Level actions do not need keys for decryption. E.g. if you do a repo-only check, it won't need to decrypt data and will just check the crc32 of all objects stored in the repo. If you start a transaction, write objects to the repo, but do not commit the transaction, borg will rollback everything to the previous transaction the next time you use the repo. But due to the lost files your case seems more problematic than just a rolled back transaction (which should be no problem at all and definitely not cause loss of committed data). > Thank you for the explanation. Is there any way to check that a) the > process is not stuck b) to get any estimate where is it now and how long > still to go? Check if there is borg client (or borg repo server) activity, data flowing to disk or via network. If you have an idea how big the initial backup was and how long it took, you can use that as the basis for a rough estimate. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From geek at airmail.cc Mon Apr 23 13:12:20 2018 From: geek at airmail.cc (geek at airmail.cc) Date: Mon, 23 Apr 2018 19:12:20 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> Message-ID: On 2018-04-23 19:07, Thomas Waldmann wrote: >> Thank you for the explanation. Is there any way to check that a) the >> process is not stuck b) to get any estimate where is it now and how >> long still to go? > Check if there is borg client (or borg repo server) activity, data > flowing to disk or via network. It's all running locally. How can I do that? > If you have an idea how big the initial backup was and how long it > took, you can use that as the basis for a rough estimate. Nope. But as I told the repository check (CRC32 check) was finished within 4 hours. From mitsch0087 at googlemail.com Mon Apr 23 13:54:08 2018 From: mitsch0087 at googlemail.com (Michael Sulzer) Date: Mon, 23 Apr 2018 19:54:08 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> Message-ID: Am 23.04.2018 um 19:12 schrieb geek at airmail.cc: > On 2018-04-23 19:07, Thomas Waldmann wrote: > >>> Thank you for the explanation. Is there any way to check that a) the >>> process is not stuck b) to get any estimate where is it now and how >>> long still to go? > >> Check if there is borg client (or borg repo server) activity, data >> flowing to disk or via network. > > It's all running locally. How can I do that? Have a look at the Windows Resource Monitor or if you are running Linux maybe iotop would be the solution to figure out if there is disk activity. > >> If you have an idea how big the initial backup was and how long it >> took, you can use that as the basis for a rough estimate. > > Nope. But as I told the repository check (CRC32 check) was finished > within 4 hours. > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From geek at airmail.cc Mon Apr 23 14:16:08 2018 From: geek at airmail.cc (geek at airmail.cc) Date: Mon, 23 Apr 2018 20:16:08 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> Message-ID: <71ec7f0f786cbbd2917a52fdcda52405@airmail.cc> >> It's all running locally. How can I do that? > Have a look at the Windows Resource Monitor or if you are running > Linux maybe iotop would be the solution to figure out if there is disk > activity. I am on MacOS. Sorry didn't do a research myself first. Started Acitivty Monitor. It says that current borg process already written 1.14 TB of data and saved 27.56 GB... If this is true, than it has almost read the entire data 3 times now... And what it saves at that volume? It seems that it keeps both reading & writing now... >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From t.guillet at gmail.com Tue Apr 24 08:41:59 2018 From: t.guillet at gmail.com (Thomas Guillet) Date: Tue, 24 Apr 2018 14:41:59 +0200 Subject: [Borgbackup] Modified ("M") file status with --list In-Reply-To: References: <6d8cbed7-e73e-b137-5cfc-e3dcc10da229@waldmann-edv.de> Message-ID: Hi Roland, Could it be this? https://borgbackup.readthedocs.io/en/stable/faq.html#i-am-seeing-a-added-status-for-an-unchanged-file If that file is the last file you changed (ie most recent timestamp in the whole backup), then my understanding is that borg will just flag it as "A" to avoid mtime granularity problems. Cheers, Thomas On 24 April 2018 at 14:35, wrote: > I updated to borg 1.1.5 , did another backup run, changed a file in the sourcedir and then did an additional backup run - and instead of "M" i'm still seeing "A" for the modified file > > this is my backup command: > > cd /btrfspool/backup-tmp/adminstation > > /backup/bin/borg-1.1.5 create --filter=AME --info --list --stats --numeric-owner --compression lz4 /iscsi/lun1/borg-repos/adminstation::archive-2018-04-24_1428 . > > regards > roland > > >> Gesendet: Montag, 26. Februar 2018 um 11:48 Uhr >> Von: "Thomas Guillet" >> An: borgbackup at python.org >> Betreff: Re: [Borgbackup] Modified ("M") file status with --list >> >> Hi Thomas, >> >> On 26 February 2018 at 08:58, Thomas Waldmann wrote: >> > Can you try that?: >> > >> > https://github.com/borgbackup/borg/pull/3633 >> >> This seems to fix it, thanks. I tested your PR with the simple test >> script from my original question, and also with an actual backup repo. >> I now indeed see files marked as M, and from a quick cursory look at >> the files list, the status flags (M vs A) look reasonable. >> >> Thanks a lot for looking into this! >> >> Thomas >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> From devzero at web.de Tue Apr 24 08:35:13 2018 From: devzero at web.de (devzero at web.de) Date: Tue, 24 Apr 2018 14:35:13 +0200 Subject: [Borgbackup] Modified ("M") file status with --list In-Reply-To: References: <6d8cbed7-e73e-b137-5cfc-e3dcc10da229@waldmann-edv.de> Message-ID: I updated to borg 1.1.5 , did another backup run, changed a file in the sourcedir and then did an additional backup run - and instead of "M" i'm still seeing "A" for the modified file this is my backup command: cd /btrfspool/backup-tmp/adminstation /backup/bin/borg-1.1.5 create --filter=AME --info --list --stats --numeric-owner --compression lz4 /iscsi/lun1/borg-repos/adminstation::archive-2018-04-24_1428 . regards roland > Gesendet: Montag, 26. Februar 2018 um 11:48 Uhr > Von: "Thomas Guillet" > An: borgbackup at python.org > Betreff: Re: [Borgbackup] Modified ("M") file status with --list > > Hi Thomas, > > On 26 February 2018 at 08:58, Thomas Waldmann wrote: > > Can you try that?: > > > > https://github.com/borgbackup/borg/pull/3633 > > This seems to fix it, thanks. I tested your PR with the simple test > script from my original question, and also with an actual backup repo. > I now indeed see files marked as M, and from a quick cursory look at > the files list, the status flags (M vs A) look reasonable. > > Thanks a lot for looking into this! > > Thomas > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From geek at airmail.cc Wed Apr 25 05:55:04 2018 From: geek at airmail.cc (Geek) Date: Wed, 25 Apr 2018 11:55:04 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: <71ec7f0f786cbbd2917a52fdcda52405@airmail.cc> References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> <71ec7f0f786cbbd2917a52fdcda52405@airmail.cc> Message-ID: <1099515649.20180425115504@airmail.cc> After 120 hours the Manifest rebuilding is finished: Manifest rebuild complete. 6369923 orphaned objects found! It still runs though and I have no clue what it does... Rebuilding archives? The process seems to be idle at the moment... >>> It's all running locally. How can I do that? >> Have a look at the Windows Resource Monitor or if you are running >> Linux maybe iotop would be the solution to figure out if there is disk >> activity. > I am on MacOS. Sorry didn't do a research myself first. > Started Acitivty Monitor. It says that current borg process already > written 1.14 TB of data and saved 27.56 GB... > If this is true, than it has almost read the entire data 3 times now... > And what it saves at that volume? > It seems that it keeps both reading & writing now... >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Best regards, Geek From tw at waldmann-edv.de Wed Apr 25 06:10:46 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 25 Apr 2018 12:10:46 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: <1099515649.20180425115504@airmail.cc> References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> <71ec7f0f786cbbd2917a52fdcda52405@airmail.cc> <1099515649.20180425115504@airmail.cc> Message-ID: <3f608519-e9a7-3d19-f26c-ae6e5a0c2b4c@waldmann-edv.de> On 25.04.2018 11:55, Geek wrote: > After 120 hours the Manifest rebuilding is finished: > > Manifest rebuild complete. > 6369923 orphaned objects found! Lots of chunks that are "unused" - likely due to missing metadata chunks. > It still runs though and I have no clue what it does... Rebuilding > archives? borg check --help -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From geek at airmail.cc Wed Apr 25 06:18:17 2018 From: geek at airmail.cc (Geek) Date: Wed, 25 Apr 2018 12:18:17 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: <3f608519-e9a7-3d19-f26c-ae6e5a0c2b4c@waldmann-edv.de> References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> <71ec7f0f786cbbd2917a52fdcda52405@airmail.cc> <1099515649.20180425115504@airmail.cc> <3f608519-e9a7-3d19-f26c-ae6e5a0c2b4c@waldmann-edv.de> Message-ID: <925638187.20180425121817@airmail.cc> >> 6369923 orphaned objects found! > Lots of chunks that are "unused" - likely due to missing metadata chunks. Will it remove them all now? How big is that in GB? This is what --help says... >> It still runs though and I have no clue what it does... Rebuilding >> archives? > borg check --help RTFMed, thank you. Lack of progress indication nor any sign of starting the next step is a bit confusing. From geek at airmail.cc Thu Apr 26 01:59:34 2018 From: geek at airmail.cc (Geek) Date: Thu, 26 Apr 2018 07:59:34 +0200 Subject: [Borgbackup] Rebuilding missing manifest - how long / Progress missing? In-Reply-To: <3f608519-e9a7-3d19-f26c-ae6e5a0c2b4c@waldmann-edv.de> References: <8461b062b98a657179e25bec62149142@airmail.cc> <41729429-1683-7661-ad83-df0de4d9bc25@waldmann-edv.de> <71ec7f0f786cbbd2917a52fdcda52405@airmail.cc> <1099515649.20180425115504@airmail.cc> <3f608519-e9a7-3d19-f26c-ae6e5a0c2b4c@waldmann-edv.de> Message-ID: <1862884931.20180426075934@airmail.cc> Hello Thomas, It stays 2 days on this message now: >> Manifest rebuild complete. >> 6369923 orphaned objects found! Disk activity is completely idle, so it is not working with the disk at all. Is it normal? Thanks > On 25.04.2018 11:55, Geek wrote: >> After 120 hours the Manifest rebuilding is finished: >> >> Manifest rebuild complete. >> 6369923 orphaned objects found! > Lots of chunks that are "unused" - likely due to missing metadata chunks. >> It still runs though and I have no clue what it does... Rebuilding >> archives? > borg check --help From ndbecker2 at gmail.com Thu Apr 26 08:03:13 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 26 Apr 2018 12:03:13 +0000 Subject: [Borgbackup] Option for no cross-device backup? Message-ID: Is there a --no-cross-device option? I accidentally ran borg when I had a (huge) remote server mounted, and borg obediently attempted to back it all up -------------- next part -------------- An HTML attachment was scrubbed... URL: From clickwir at gmail.com Thu Apr 26 08:54:15 2018 From: clickwir at gmail.com (Zack Coffey) Date: Thu, 26 Apr 2018 06:54:15 -0600 Subject: [Borgbackup] Option for no cross-device backup? In-Reply-To: References: Message-ID: https://borgbackup.readthedocs.io/en/stable/usage/create.html Try the -x option? On Thu, Apr 26, 2018 at 6:03 AM, Neal Becker wrote: > Is there a --no-cross-device option? > > I accidentally ran borg when I had a (huge) remote server mounted, and > borg obediently attempted to back it all up > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gait at ATComputing.nl Tue May 1 03:20:42 2018 From: gait at ATComputing.nl (Gerrit A. Smit) Date: Tue, 1 May 2018 09:20:42 +0200 Subject: [Borgbackup] Duration: 16 hours 16.27 seconds Message-ID: Hello, It would be nice if minutes would be display, even when zero. -- Met vriendelijke groeten, AT COMPUTING BV Gerrit A. Smit AT Computing Telefoon: +31 24 352 72 22 D? one-stop-Linux-shop Telefoon cursussecretariaat: +31 24 352 72 72 Kerkenbos 12-38 TI at ATComputing.nl 6546 BE Nijmegen www.atcomputing.nl https://www.linkedin.com/in/gesmit -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at gasaway.org Thu May 10 12:20:49 2018 From: dave at gasaway.org (David Gasaway) Date: Thu, 10 May 2018 09:20:49 -0700 Subject: [Borgbackup] Segment size Message-ID: Hi, A couple of years ago(1), the default segment size was changed from ~5MiB to ~500MiB. Can anyone provide more detail on the reasoning? I am planning to use an s3ql filesystem as storage, and am concerned the large segments will create headaches. If the change was to improve disk I/O performance, that wouldn't really apply to this use case (network is the bottleneck, anyway). I'd like to know if there are any other consequences to a much smaller segment size. Thanks. (1) https://github.com/borgbackup/borg/commit/c52861e0cafafa2988ae98c3e25918068463012d -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Thu May 10 13:12:11 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 10 May 2018 19:12:11 +0200 Subject: [Borgbackup] Segment size In-Reply-To: References: Message-ID: > A couple of years ago(1), the default segment size was changed from > ~5MiB to ~500MiB.? Can anyone provide more detail on the reasoning?? I > am planning to use an s3ql filesystem as storage, and am concerned the > large segments will create headaches.? If the change was to improve disk > I/O performance, that wouldn't really apply to this use case (network is > the bottleneck, anyway).? I'd like to know if there are any other > consequences to a much smaller segment size. IIRC it was mainly about performance (when using a local disk for the repo). Maybe you can find more infos in some (closed?) github issue. Please note that this change was just about the DEFAULT segment size, you can change it in the repo config if you prefer some other size. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From dave at gasaway.org Thu May 10 15:24:35 2018 From: dave at gasaway.org (David Gasaway) Date: Thu, 10 May 2018 12:24:35 -0700 Subject: [Borgbackup] Segment size In-Reply-To: References: Message-ID: On Thu, May 10, 2018 at 10:12 AM, Thomas Waldmann wrote: > > IIRC it was mainly about performance (when using a local disk for the > repo). > > Maybe you can find more infos in some (closed?) github issue. > > Please note that this change was just about the DEFAULT segment size, > you can change it in the repo config if you prefer some other size. > Yes, I found out about max_segment_size through a couple of closed issues before I posted. Those were people wanting to *increase* the value, not *decrease* it. So far, sounding like it would be pretty safe to go smaller. Thanks. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From piter8111 at gmail.com Tue May 15 07:53:23 2018 From: piter8111 at gmail.com (Piotr K) Date: Tue, 15 May 2018 13:53:23 +0200 Subject: [Borgbackup] Issue with delete old backups from few full backups Message-ID: Hello, I have logical issue. I need delete older backup than 1 months. Unfourtanetly I use few repository/full backup. For exmaple I have below repository with daily backups: hostname.com/20180407 hostname.com/20180414 hostname.com/20180421 hostname.com/20180428 When I use command: /usr/local/bin/borg prune -v ssh://borgbackup at hostname.com/20180407 -H 1 -d 7 -w 4 -m 1 -y 0 --list some backup willn't deleted becouse will not create new backups. I know that is correctly behaviour. My question is: Is there a possibility deleted all old backups form few repository with one borg comand? Repository I will delete with "borg delete". I try find good solve this issue. I will be grateful for your help Regards -------------- next part -------------- An HTML attachment was scrubbed... URL: From mh+borgbackup at zugschlus.de Tue May 15 15:39:47 2018 From: mh+borgbackup at zugschlus.de (Marc Haber) Date: Tue, 15 May 2018 21:39:47 +0200 Subject: [Borgbackup] hashindex fopen for reading failed Message-ID: <20180515193947.nb5d7hm7qlgopw6x@torres.zugschlus.de> Hi, I recently had a host crash while a borg create was running ("client side"), and after rebooting, the borg create won't resume. Obviously, one of the cache file is gone, so borg create aborts with $ time sudo -H borg create --verbose --progress --exclude-from /etc/borgbackup-exclude ssh://borgbackup//media/borgbackup/backup/bor g/$(hostname)/system::$(date +%Y%m%d)b / hashindex: /root/.cache/borg/cb4113cc184e0481d17fe78f342e855705f9855af8a169100146d98a3628f96a/chunks: fopen for reading failed (No such file or directory) Local Exception. Traceback (most recent call last): File "/usr/lib/python3/dist-packages/borg/archiver.py", line 2052, in main exit_code = archiver.run(args) File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1997, in run return func(args) File "/usr/lib/python3/dist-packages/borg/archiver.py", line 90, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/lib/python3/dist-packages/borg/archiver.py", line 299, in do_create with Cache(repository, key, manifest, do_files=args.cache_files, lock_wait=self.lock_wait) as cache: File "/usr/lib/python3/dist-packages/borg/cache.py", line 73, in __init__ self.open(lock_wait=lock_wait) File "/usr/lib/python3/dist-packages/borg/cache.py", line 185, in open self.rollback() File "/usr/lib/python3/dist-packages/borg/cache.py", line 267, in rollback self._do_open() File "/usr/lib/python3/dist-packages/borg/cache.py", line 178, in _do_open self.chunks = ChunkIndex.read(os.path.join(self.path, 'chunks').encode('utf-8')) File "borg/hashindex.pyx", line 79, in borg.hashindex.IndexBase.read (borg/hashindex.c:1550) File "borg/hashindex.pyx", line 67, in borg.hashindex.IndexBase.__cinit__ (borg/hashindex.c:1353) Exception: hashindex_read failed Platform: Linux 4.16.8-zgsrv20080 #3 SMP PREEMPT Wed May 9 10:50:48 UTC 2018 x86_64 Linux: debian 9.4 Borg: 1.0.9 Python: CPython 3.5.3 PID: 14685 CWD: /home/mh sys.argv: ['/usr/bin/borg', 'create', '--verbose', '--progress', '--exclude-from', '/etc/borgbackup-exclude', 'ssh://borgbackup//media/borgbackup/backup/borg/hostname/system::20180515b', '/'] SSH_ORIGINAL_COMMAND: None real 0m2,212s user 0m0,686s sys 0m0,040s $ This repeats when repeated. How do I get out of this mess and get a backup? Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 From tw at waldmann-edv.de Tue May 15 17:22:00 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 15 May 2018 23:22:00 +0200 Subject: [Borgbackup] hashindex fopen for reading failed In-Reply-To: <20180515193947.nb5d7hm7qlgopw6x@torres.zugschlus.de> References: <20180515193947.nb5d7hm7qlgopw6x@torres.zugschlus.de> Message-ID: <7293cc9d-f62a-ba10-2315-b25b2e834696@waldmann-edv.de> Hi Marc, > I recently had a host crash while a borg create was running ("client > side"), and after rebooting, the borg create won't resume. Obviously, > one of the cache file is gone, so borg create aborts with > > $ time sudo -H borg create --verbose --progress --exclude-from /etc/borgbackup-exclude ssh://borgbackup//media/borgbackup/backup/bor > g/$(hostname)/system::$(date +%Y%m%d)b / > hashindex: /root/.cache/borg/cb4113cc184e0481d17fe78f342e855705f9855af8a169100146d98a3628f96a/chunks: fopen for reading failed (No such file or directory) Yeah, looks like the chunks cache is gone. Can you please do a: ls -lR /root/.cache/borg/cb4113cc184e04* > Borg: 1.0.9 Just saying "there were some bugfix releases after that", but I know "Debian Stable" policy. Also not sure whether this was somehow improved. > This repeats when repeated. > > How do I get out of this mess and get a backup? You could kill the local caches at any time, they contain no information that could not be rebuilt from the repo (in the case of the chunks index. Or from local fs, in the case of the files cache). But, for big repos this can take quite a while, so let's first see if there's a better way. Is this the same incident that also has corrupted your files cache (you added a comment under the PR on github recently)? Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mario at emmenlauer.de Tue May 15 17:20:09 2018 From: mario at emmenlauer.de (Mario Emmenlauer) Date: Tue, 15 May 2018 23:20:09 +0200 Subject: [Borgbackup] extended prune In-Reply-To: References: Message-ID: Dear Borg Developers, I'm coming back to a really old thread here. A long time ago I added a minor patch to borg prune, that enabled passing an explicit list of archives to be pruned. In the meantime I have not further followed up the topic, but I am still very fond of the explicit purge list. Is this something that may go into official borg? The reason to have this option is that while borg has quite powerful selection options for prune, they do not cover more complex cases like pruning based on size, or combining prune by time and archive name. Power users can often generate the list of archives to prune with external tools. This new option allows them to specify such a hand-crafted list for borg prune. See below for the patch and more details. Would it be possible to add this to borg? All the best, Mario On 23.12.2016 23:53, Mario Emmenlauer wrote: > > Dear All, > > it seems I am pretty lucky this year to have an early Christmas > present. First, I found the large archives in my repo by pure > chance and could free 50% of disk space with only a few deletions. > Now the actual disk usage is at 1.8TB again, which matches borg's > report of deduplicated size. > > Furthermore, it seems that those huge deletions where the only > "slow" ones, because later I could prune another ~200 of ~500 > archives in just little over 10 minutes, with borg 1.0.9. > > Finally, I seemed unable to get prune do exactly what I hoped for. > It might be me, but I did not find exactly the right combination of > options. I take backups once per week, and if they are older than > one year, I'd like to keep only every other week. > In any case I was also curious to enable prune to handle a manual > selection of archives, so I tried, and got it working pretty easily. > I extended archive.py and helper.py with two new prune options > --keep-list and --remove-list, where the former takes a list of > archives to keep (all others are pruned) and the latter takes a > list of archives to prune (all others are kept). My patch against > borg 1.0.9 is available here > https://github.com/emmenlau/borg/tree/emmenlau_better_prune > and I'm happy to make a PR if anyone is interested (sorry for the > bold name, its really just a very minor extension to prune). > > > Finally, thanks a lot again for the very nice borg! Your code was > very easy to read, and I found very helpful compile instructions > in the readme! This allowed me to get productive within a few > minutes! Nice work! It would be awesome to add the pyinstaller > instructions to the readme, but they where sufficiently easy to > find in an github issue report. > > Thanks, and happy holidays, > > Mario Emmenlauer -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 M?nchen http://www.biodataanalysis.de/ From tw at waldmann-edv.de Tue May 15 17:49:24 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 15 May 2018 23:49:24 +0200 Subject: [Borgbackup] extended prune In-Reply-To: References: Message-ID: <8391729a-f129-3313-c1cc-5fa2093b17c5@waldmann-edv.de> Hi Mario, can you (carefully) check if borg 1.1.x already does what you want: borg delete repo archive1 archive2 archive3 ... Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mh+borgbackup at zugschlus.de Wed May 16 02:47:19 2018 From: mh+borgbackup at zugschlus.de (Marc Haber) Date: Wed, 16 May 2018 08:47:19 +0200 Subject: [Borgbackup] hashindex fopen for reading failed In-Reply-To: <7293cc9d-f62a-ba10-2315-b25b2e834696@waldmann-edv.de> References: <20180515193947.nb5d7hm7qlgopw6x@torres.zugschlus.de> <7293cc9d-f62a-ba10-2315-b25b2e834696@waldmann-edv.de> Message-ID: <20180516064719.e2322allb2ye4tlz@torres.zugschlus.de> On Tue, May 15, 2018 at 11:22:00PM +0200, Thomas Waldmann wrote: > > Borg: 1.0.9 > > Just saying "there were some bugfix releases after that", but I know > "Debian Stable" policy. Also not sure whether this was somehow improved. Debian's release policy does not need improving, it's fine as it is. I do understand this causes constant annoyance with Developers, but I am an operations guy and I love systems that don't change every other week. > > This repeats when repeated. > > > > How do I get out of this mess and get a backup? > > You could kill the local caches at any time, they contain no information > that could not be rebuilt from the repo (in the case of the chunks > index. Or from local fs, in the case of the files cache). Good to know that. Will zap the local cache, no need to investigate these issues on an outdated version. > Is this the same incident that also has corrupted your files cache (you > added a comment under the PR on github recently)? No, different client, but same server (both running 1.1.5). Greetings Marc -- ----------------------------------------------------------------------------- Marc Haber | "I don't trust Computers. They | Mailadresse im Header Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402 Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421 From mario at emmenlauer.de Wed May 16 07:55:23 2018 From: mario at emmenlauer.de (Mario Emmenlauer) Date: Wed, 16 May 2018 13:55:23 +0200 Subject: [Borgbackup] extended prune In-Reply-To: <8391729a-f129-3313-c1cc-5fa2093b17c5@waldmann-edv.de> References: <8391729a-f129-3313-c1cc-5fa2093b17c5@waldmann-edv.de> Message-ID: <5154fad2-4af7-4d41-e359-2c6f44002ffe@emmenlauer.de> Dear Thomas, oh my, this sounds great! Do I understand correctly that this kind of deletion has the same I/O benefits as prune would have? And about the invocation syntax, does the following example look correct to you? So for multi-deletion, there is no double colon between repo and archive, correct? #> borg delete /path/to/repo hostA-2018051646 hostB-2018040250 That confused me a bit in the docs, sorry! All the best, Mario On 15.05.2018 23:49, Thomas Waldmann wrote: > Hi Mario, > > can you (carefully) check if borg 1.1.x already does what you want: > > borg delete repo archive1 archive2 archive3 ... > > Cheers, Thomas > -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 M?nchen http://www.biodataanalysis.de/ From tw at waldmann-edv.de Wed May 16 11:11:23 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 16 May 2018 17:11:23 +0200 Subject: [Borgbackup] extended prune In-Reply-To: <5154fad2-4af7-4d41-e359-2c6f44002ffe@emmenlauer.de> References: <8391729a-f129-3313-c1cc-5fa2093b17c5@waldmann-edv.de> <5154fad2-4af7-4d41-e359-2c6f44002ffe@emmenlauer.de> Message-ID: > oh my, this sounds great! Do I understand correctly that this > kind of deletion has the same I/O benefits as prune would have? No, in the code it is just deleting / committing each given archive individually. So you just save entering multiple borg commands. > And about the invocation syntax, does the following example > look correct to you? So for multi-deletion, there is no double > colon between repo and archive, correct? > > #> borg delete /path/to/repo hostA-2018051646 hostB-2018040250 Yes. But the code would even understand both: # borg delete repo archive1 archive2 archive3 # borg delete repo::archive1 archive2 archive3 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mario at emmenlauer.de Wed May 16 11:35:12 2018 From: mario at emmenlauer.de (Mario Emmenlauer) Date: Wed, 16 May 2018 17:35:12 +0200 Subject: [Borgbackup] extended prune In-Reply-To: References: <8391729a-f129-3313-c1cc-5fa2093b17c5@waldmann-edv.de> <5154fad2-4af7-4d41-e359-2c6f44002ffe@emmenlauer.de> Message-ID: Thanks again for the help! Below more: On 16.05.2018 17:11, Thomas Waldmann wrote: >> oh my, this sounds great! Do I understand correctly that this >> kind of deletion has the same I/O benefits as prune would have? > > No, in the code it is just deleting / committing each given archive > individually. > > So you just save entering multiple borg commands. Ok, I see! In my understanding, the actual removal of chunks can be more efficient if performed in one go, is that correct? In my typical use case, I prune only once every hundred backups or so, so I assume I can win a *lot* of performance and reduced disk stress from a single pass deletion. I guess that is what prune does? Then it would be great to still have a prune option that takes a list, or alternatively a deletion that uses a separate garbage collection step. Would one of these two be a reasonable extension to borg? All the best, Mario Emmenlauer -- BioDataAnalysis GmbH, Mario Emmenlauer Tel. Buero: +49-89-74677203 Balanstr. 43 mailto: memmenlauer * biodataanalysis.de D-81669 M?nchen http://www.biodataanalysis.de/ From tw at waldmann-edv.de Wed May 16 12:02:57 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 16 May 2018 18:02:57 +0200 Subject: [Borgbackup] extended prune In-Reply-To: References: <8391729a-f129-3313-c1cc-5fa2093b17c5@waldmann-edv.de> <5154fad2-4af7-4d41-e359-2c6f44002ffe@emmenlauer.de> Message-ID: <5659f133-e440-9139-9e33-50791a8f4ba0@waldmann-edv.de> > In my understanding, the actual removal of chunks can be more efficient > if performed in one go, is that correct? Yes. borg prune does it in one go, but borg delete currently doesn't. > Then it would be great to still have a prune option that takes a list, No, let's keep borg prune as a "policy based deletion tool" (usually time-based policy) and let's use borg delete if you already know what to delete (e.g. by archive names). borg prune is already confusing/complex enough, so we should not make it even more complex. > or alternatively a deletion that uses a separate garbage collection step. Yes, I think changing do_delete / _delete_archives is the way to go. If we can do it quick (with less commits) in do_prune, we also can do that for deletion. See archiver.py. > Would one of these two be a reasonable extension to borg? Yes, PRs welcome. Something also missing for borg delete (and maybe a good idea to implement FIRST) is a --dry-run option. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From dave at gasaway.org Wed May 16 13:44:28 2018 From: dave at gasaway.org (David Gasaway) Date: Wed, 16 May 2018 10:44:28 -0700 Subject: [Borgbackup] ctime vs. mtime Message-ID: Hi, I implemented a new backup with borg that was taking longer than expected - 2 hours compared to 2 minutes for the rdiff-backup equivalent. I traced the problem down ctime. There is a script that runs nightly that applied 'chown', 'chmod' and 'setfacl' to the bulk of the filesystem to reset permissions to desired values regardless where the data came from (NFS, SAMBA, local). Unsurprisingly, this updates ctime even if there is no actual change in permissions in the end. First, I would be thrilled if anyone could point me to software that could do the above while only only updating ctime if there is a change necessary. I'm not keen on trying to write something myself. Second, I've read the 'borg create' docs and the discussion on issue #911 with respect to ctime vs. mtime, but I want to ask a couple of things. 1) If there is a permission-only change (ctime updated, mtime unchanged), will the backup have the modified permissions? 2) Any potential consequences of changing from ctime to mtime in an existing repo? Should I delete the files cache to force a rebuild from scratch? Thanks. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Wed May 16 18:04:36 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 17 May 2018 00:04:36 +0200 Subject: [Borgbackup] ctime vs. mtime In-Reply-To: References: Message-ID: > I implemented a new backup with borg that was taking longer than expected - > 2 hours compared to 2 minutes for the rdiff-backup equivalent. I traced > the problem down ctime. There is a script that runs nightly that applied > 'chown', 'chmod' and 'setfacl' to the bulk of the filesystem to reset > permissions to desired values regardless where the data came from (NFS, > SAMBA, local). Unsurprisingly, this updates ctime even if there is no > actual change in permissions in the end. Yeah, as expected. But that somehow sounds like a dirty hack anyway, so maybe you could just do that differently. > First, I would be thrilled if anyone could point me to software that could > do the above while only only updating ctime if there is a change > necessary. I'm not keen on trying to write something myself. ctime always gets updated if you do chown & co. But you could maybe use "find" to find files that do not have the desired owner and then chown only these. Or just use ACLs to get the desired effect. You can also change borg create to use mtime,inode,size instead of ctime,inode,size - but that is less safe when somebody/something does weird things with the mtime, see changelog and issue tracker. > Second, I've read the 'borg create' docs and the discussion on issue #911 > with respect to ctime vs. mtime, but I want to ask a couple of things. Yes, that's the ticket I meant. > 1) If there is a permission-only change (ctime updated, mtime unchanged), > will the backup have the modified permissions? borg ALWAYS reads the current owner/group/mode/acls/xattrs from the filesystem and stores it into the archive it creates. The change detection only controls whether the file contents get read, chunked, hashed, ... > 2) Any potential consequences of changing from ctime to mtime in an > existing repo? Guess it should work ok. borg checks for the stored timestamp being same as the timestamp read from the fs. If it is SAME and also the other stuff (inode no and size) matches, the file is considered unchanged. So is does not matter whether a timestamp is newer or older, if it is not the same, that would trigger in both cases. > Should I delete the files cache to force a rebuild from scratch? That would work for sure. In any case, expect a slow first backup when doing that. Cheers, Thomas -- Waldmann EDV Systeme/Service GbR Gerhard Waldmann, Thomas Waldmann Pleidelsheimer Str. 25 74321 Bietigheim-Bissingen Weitere Infos siehe: http://waldmann-edv.de/ GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From dave at gasaway.org Wed May 16 19:50:18 2018 From: dave at gasaway.org (David Gasaway) Date: Wed, 16 May 2018 16:50:18 -0700 Subject: [Borgbackup] ctime vs. mtime In-Reply-To: References: Message-ID: On Wed, May 16, 2018 at 3:04 PM, Thomas Waldmann wrote: > But that somehow sounds like a dirty hack anyway, so maybe you could > just do that differently. I agree it's a dirty hack. But I found it necessary after observing default permissions for files created/moved over NFS and CIFS. > But you could maybe use "find" to find files that do not have the > desired owner and then chown only these. Easy enough for the chown/chmod calls, I think. I know of no equivalent for the setfacl calls. Maybe I could hack something together with pylibacl. Doesn't sound fun. > Or just use ACLs to get the desired effect. I use ACLs, but also owner and permission bits. I set this up long ago, so could not explain why at this time. Anyway, I'd still have a script that changed ACLs and ctime. > borg ALWAYS reads the current owner/group/mode/acls/xattrs from the > filesystem and stores it into the archive it creates. > > The change detection only controls whether the file contents get read, > chunked, hashed, ... Cool, thanks. > Guess it should work ok. borg checks for the stored timestamp being same > as the timestamp read from the fs. If it is SAME and also the other > stuff (inode no and size) matches, the file is considered unchanged. > > So is does not matter whether a timestamp is newer or older, if it is > not the same, that would trigger in both cases. Okay, so it would only cause a problem if, since the prior backup, the file was modified and somehow the new mtime was exactly the same as the old ctime, in addition to inode and size. I can live with that. :) > > Should I delete the files cache to force a rebuild from scratch? > > That would work for sure. > > In any case, expect a slow first backup when doing that. On the order of 2 hours, I imagine, since almost everything is getting re-chunked and re-hashed now. I can live with that, too, but it sounds like I'd be pretty safe just changing the file cache mode. That is, as safe as mtime can be. I can always switch back to ctime once I better solve my permission issues. I appreciate your help. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org From anarcat at debian.org Tue May 29 10:21:59 2018 From: anarcat at debian.org (=?utf-8?Q?Antoine_Beaupr=C3=A9?=) Date: Tue, 29 May 2018 10:21:59 -0400 Subject: [Borgbackup] backing up borg with borg Message-ID: <874liqy22g.fsf@curie.anarc.at> Hi! I'm looking at securing a backup setup against compromised clients. In that model, clients are untrusted: they run untrusted code and might explode at any minute. The goal is to still protect their data against compromise. I am aware of various "pull" methods[900] but those are not very practical for my use case: I don't feel comfortable having a busy backup server with access on all machines. It introduces too much of a security liability. [900]: https://github.com/borgbackup/borg/issues/900 I am also aware of --append-only on the server. This is (basically) incompatible with purge as you need to audit the transaction log before running purges[2251]. Append-only mode was described elsewhere as "confusing"[3504] and not something I wish to use at this time. [2251]: https://github.com/borgbackup/borg/issues/2251 [3504]: https://github.com/borgbackup/borg/issues/3504 Instead, what I am considering is a two-stage configuration, which I've mentioned elsewhere: 1. stage one: every client pushes backups to the central server in isolated accounts, as per the [Hosting repositories][] deployment guide. Our first protection layer is the security of the client. If it is compromised, it can (a) destroy itself and its backups, but not other clients backups, unless (b) the security of the *server* is compromised. 2. stage two: an offsite server *pulls* backups from the stage one central server. Not sure yet how to do this, but it should keep historical data somehow. This works around the threats identified in 1(a) and 1(b) above: even if the client self-destroys, the stage two offsite server has history and can recover. Similarly, if a client manages to compromise the stage one server and contaminate other datasets, we can still recover. [Hosting repositories]: https://borgbackup.readthedocs.io/en/stable/deployment/hosting-repositories.html The question is: how to implement stage two. This goes back to the question of #900, broadly, but I have other options: I can use a simpler tool like [rsnapshot][] which can pull from a remote SSH serverf fairly effectively, for example. This should work well since a set of Borg archives is much simpler than a full working filesystem: it's just a bunch of blobs and is already compressed and deduplicated. [rsnapshot]: http://rsnapshot.org/ But I'm wondering if I could somehow pull using Borg instead. And this brings up the question of what would happen if you would backup a Borg repository with Borg. Then we have Inception! Borg backing up Borg! This makes my head explode and brings up interesting questions: * Would this work at all? Would it explode in a weird Hollywood movie? * Would it deduplicate correctly? * How about compression: pointless? * Which chunk size should I pick? Segments seem to be around 500MB, should I shoot for somewhere around that? Or closer to the original chunk size? * What's the state of the art for pull-mode operations these days? * Should I even bother? * Who's assimilated now, uh? ;) Thanks for any feedback. :) A. -- Il faut tout un village pour ?lever un enfant. - Proverbe africain From tw at waldmann-edv.de Sun Jun 3 14:18:05 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 3 Jun 2018 20:18:05 +0200 Subject: [Borgbackup] backing up borg with borg In-Reply-To: <874liqy22g.fsf@curie.anarc.at> References: <874liqy22g.fsf@curie.anarc.at> Message-ID: <865b23a4-a6d8-105b-86db-23ca6c02d0ea@waldmann-edv.de> This 2 stage approach only moves the hard decision from the 1st stage server to the 2nd stage server. As you noted, the borg solution for this is append-only mode, but it has this fundamental and non-trivial problem about having to know when it is safe to switch off append-only mode and run prune / realize all deletes / prunes that have already been issued. And that you have to know somehow whether the remaining stuff is in a good state and still has all you need. When you want to solve this by keeping some history on your 2nd stage server, you will have to decide the same thing there. And if you can't decide it on stage 1, you won't be able to decide it on stage 2 either. Also, you might have some unwanted error propagation from 1st to 2nd stage (which is why the FAQ rather recommends doing backups to different places in a parallel, not serial, manner). --- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From nickeforos at gmail.com Sat Jun 9 10:27:09 2018 From: nickeforos at gmail.com (Nick Gilmour) Date: Sat, 9 Jun 2018 16:27:09 +0200 Subject: [Borgbackup] delete checkpoint Message-ID: Hi all, I'm trying to delete a checkpoint this way: *$ BORG_REPO="/mnt/BACKUPS/Borg/home"* *$ borg list --short "$BORG_REPO" | grep "\.checkpoint$" | parallel --progress --jobs 1 borg delete "$BORG_REPO"::{}* lile I found here: https://github.com/borgbackup/borg/issues/998 The last lines in the output are these: *Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete* *local:1/0/100%/0.0s* but there is nothing else. I would expect to see the progress of borg deleting this file. The first time I left it runiing for some hours until I stopped it with Ctrl-C. Now I'm trying again after a reboot again the same. BTW, It complains if I include the --dry-run option. Is this normal? Shouldn't it be done within minutes? Regards, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Mon Jun 11 07:23:29 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 11 Jun 2018 13:23:29 +0200 Subject: [Borgbackup] borgbackup 1.1.6 released! Message-ID: <3f0c2e87-34bb-1de5-7e11-e2169dd4e665@waldmann-edv.de> Released borgbackup 1.1.6 with misc. bug fixes and some small feature enhancements. https://github.com/borgbackup/borg/releases/tag/1.1.6 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From ohnonot-github at posteo.de Sun Jun 17 05:33:21 2018 From: ohnonot-github at posteo.de (ohnonot-github at posteo.de) Date: Sun, 17 Jun 2018 12:33:21 +0300 Subject: [Borgbackup] the logic of deduplication Message-ID: <4cc5cf23-30a0-798b-0cc2-651c079f9453@iki.fi> Hello, I have succesfully been using borgbackup for almost a year; every week it creates (incremental,deduplicating - synonyms?) backups of my partitions. Judging from the sizes of the combined backups, the deduplication seems to be working well, i.e. after an initial full backup (large) it doesn't much grow in size. Nevertheless, I would like to remove older backups. If nothing else, the integrity checks seem to be taking more and more time. Is this possible at all? without destroying the incremental/deduplication functionality? greetings, o. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin.rampe at uni-jena.de Sun Jun 17 07:09:16 2018 From: benjamin.rampe at uni-jena.de (Benjamin Redling) Date: Sun, 17 Jun 2018 13:09:16 +0200 Subject: [Borgbackup] the logic of deduplication In-Reply-To: <4cc5cf23-30a0-798b-0cc2-651c079f9453@iki.fi> References: <4cc5cf23-30a0-798b-0cc2-651c079f9453@iki.fi> Message-ID: <102F5946-A69B-44B5-A87A-675CC2EFA55B@uni-jena.de> Hello Ohnonot, Am 17. Juni 2018 11:33:21 MESZ, schrieb ohnonot-github at posteo.de: > ... I would like to remove older backups. If nothing else, >the >integrity checks seem to be taking more and more time. > >Is this possible at all? without destroying the >incremental/deduplication functionality? What's wrong with http://borgbackup.readthedocs.io/en/stable/usage/prune.html ? Regards, Benjamin -- FSU Jena | JULIELab.de/Staff/Benjamin+Redling.html From lars-borgbackup at 12357.de Wed Jun 20 04:43:50 2018 From: lars-borgbackup at 12357.de (Lars Bendel) Date: Wed, 20 Jun 2018 10:43:50 +0200 Subject: [Borgbackup] Build Static Borg Binary fail with Vagrantfile Message-ID: <8e607709-c918-3f33-ed63-bdc150b77ab2@12357.de> Hello List, i'd like to backup my Centos7 and Debian Stretch Systems with borgbackup. Because the versions from the Repositorys are different and old, i decided to build the v1.1.6 with the Vagrantfile as static binary, but that's failed with the error below for both Vagrant boxes (centos7_64 & stretch64). Is there any step i forget or should i done befor i starting the boxes with "vagrant up centos_64" or? "vagrant up stretch64"? In the doc's i didn't found any solution and an update of pip to version 10.0.1 doesn't help. best regards ?? Lars Bendel <----- SNIP centos7_64 -----> ... ??? centos7_64: Building wheels for collected packages: py-cpuinfo ??? centos7_64:?? Running setup.py bdist_wheel for py-cpuinfo: started ??? centos7_64:?? Running setup.py bdist_wheel for py-cpuinfo: finished with status 'done' ??? centos7_64:?? Stored in directory: /home/vagrant/.cache/pip/wheels/bd/e6/74/5f9ff2150a62868acf69169db535d99949f838ac2f8e0b5d4b ??? centos7_64: Successfully built py-cpuinfo ??? centos7_64: Installing collected packages: setuptools-scm, virtualenv, six, pluggy, py, tox, attrs, more-itertools, atomicwrites,pytest, pytest-forked, apipkg, execnet, pytest-xdist, coverage, pytest-cov, py-cpuinfo, pytest-benchmark, Cython ??? centos7_64: Successfully installed Cython-0.28.3 apipkg-1.4 atomicwrites-1.1.5 attrs-18.1.0 coverage-4.5.1 execnet-1.5.0 more-itertools-4.2.0 pluggy-0.6.0 py-1.5.3 py-cpuinfo-4.0.0 pytest-3.6.1 pytest-benchmark-3.1.1 pytest-cov-2.5.1 pytest-forked-0.2 pytest-xdist-1.22.2 setuptools-scm-2.1.0 six-1.11.0 tox-3.0.0 virtualenv-16.0.0 ??? centos7_64: You are using pip version 9.0.1, however version 10.0.1 is available. ??? centos7_64: You should consider upgrading via the 'pip install --upgrade pip' command. ??? centos7_64: Detected and preferring liblz4 over bundled LZ4 ??? centos7_64: Traceback (most recent call last): ??? centos7_64:?? File "setup.py", line 856, in ??? centos7_64:???? extras_require=extras_require, ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/lib/python3.5/distutils/core.py", line 108, in setup ??? centos7_64:???? _setup_distribution = dist = klass(attrs) ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 318, in __init__ ??? centos7_64:???? _Distribution.__init__(self, attrs) ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/lib/python3.5/distutils/dist.py", line 281, in __init__ ??? centos7_64:???? self.finalize_options() ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 376, in finalize_options ??? centos7_64:???? ep.load()(self, ep.name, value) ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/integration.py",line 22, in version_keyword ??? centos7_64:???? dist.metadata.version = get_version(**value) ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 119, in get_version ??? centos7_64:???? parsed_version = _do_parse(root, parse) ??? centos7_64:?? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 97, in _do_parse ??? centos7_64:???? "use git+https://github.com/user/proj.git#egg=proj" % root) ??? centos7_64: LookupError: setuptools-scm was unable to detect version for '/vagrant/borg/borg'. ??? centos7_64: ??? centos7_64: Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work. ??? centos7_64: ??? centos7_64: For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj ??? centos7_64: Obtaining file:///vagrant/borg/borg ??? centos7_64:???? Complete output from command python setup.py egg_info: ??? centos7_64:???? Detected and preferring liblz4 over bundled LZ4 ??? centos7_64:???? Traceback (most recent call last): ??? centos7_64:?????? File "", line 1, in ??? centos7_64:?????? File "/vagrant/borg/borg/setup.py", line 856, in ??? centos7_64:???????? extras_require=extras_require, ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/lib/python3.5/distutils/core.py", line 108, in setup ??? centos7_64:???????? _setup_distribution = dist = klass(attrs) ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 318, in __init__ ??? centos7_64:???????? _Distribution.__init__(self, attrs) ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/lib/python3.5/distutils/dist.py", line 281, in __init__ ??? centos7_64:???????? self.finalize_options() ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 376, in finalize_options ??? centos7_64:???????? ep.load()(self, ep.name, value) ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/integration.py", line 22, in version_keyword ??? centos7_64:???????? dist.metadata.version = get_version(**value) ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 119, in get_version ??? centos7_64:???????? parsed_version = _do_parse(root, parse) ??? centos7_64:?????? File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 97, in _do_parse ??? centos7_64:???????? "use git+https://github.com/user/proj.git#egg=proj" % root) ??? centos7_64:???? LookupError: setuptools-scm was unable to detect version for '/vagrant/borg/borg'. ??? centos7_64: ??? centos7_64:???? Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such asGitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work. ??? centos7_64: ??? centos7_64:???? For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj ??? centos7_64: ??? centos7_64:???? ---------------------------------------- ??? centos7_64: Command "python setup.py egg_info" failed with error code 1 in /vagrant/borg/borg/ ??? centos7_64: You are using pip version 9.0.1, however version 10.0.1 is available. ??? centos7_64: You should consider upgrading via the 'pip install --upgrade pip' command. <----- /SNIP centos7_64 -----> <----- SNIP stretch64 -----> ... ??? stretch64: Building wheels for collected packages: Cython, coverage, py-cpuinfo ??? stretch64:?? Running setup.py bdist_wheel for Cython ??? stretch64:?? Stored in directory: /home/vagrant/.cache/pip/wheels/ab/03/d5/599875adc6902e7bdd4fd2da755db99987130a25d30314a162 ??? stretch64:?? Running setup.py bdist_wheel for coverage ??? stretch64:?? Stored in directory: /home/vagrant/.cache/pip/wheels/24/76/26/c60de6a591b00f71dcf87941eb41b8f29820857eade91fe66f ??? stretch64:?? Running setup.py bdist_wheel for py-cpuinfo ??? stretch64:?? Stored in directory: /home/vagrant/.cache/pip/wheels/bd/e6/74/5f9ff2150a62868acf69169db535d99949f838ac2f8e0b5d4b ??? stretch64: Successfully built Cython coverage py-cpuinfo ??? stretch64: Installing collected packages: setuptools-scm, virtualenv, pluggy, six, py, tox, atomicwrites, attrs, more-itertools, pytest, apipkg, execnet, pytest-forked, pytest-xdist, coverage, pytest-cov, py-cpuinfo, pytest-benchmark, Cython ??? stretch64: Successfully installed Cython-0.28.3 apipkg-1.4 atomicwrites-1.1.5 attrs-18.1.0 coverage-4.5.1 execnet-1.5.0 more-itertools-4.2.0 pluggy-0.6.0 py-1.5.3 py-cpuinfo-4.0.0 pytest-3.6.1 pytest-benchmark-3.1.1 pytest-cov-2.5.1 pytest-forked-0.2 pytest-xdist-1.22.2 setuptools-scm-2.1.0 six-1.11.0 tox-3.0.0 virtualenv-16.0.0 ??? stretch64: You are using pip version 7.1.2, however version 10.0.1 is available. ??? stretch64: You should consider upgrading via the 'pip install --upgrade pip' command. ??? stretch64: Detected and preferring liblz4 over bundled LZ4 ??? stretch64: Traceback (most recent call last): ??? stretch64:?? File "setup.py", line 856, in ??? stretch64:???? extras_require=extras_require, ??? stretch64:?? File "/usr/lib/python3.5/distutils/core.py", line 108, in setup ??? stretch64:???? _setup_distribution = dist = klass(attrs) ??? stretch64:?? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 272, in __init__ ??? stretch64:???? _Distribution.__init__(self,attrs) ??? stretch64:?? File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__ ??? stretch64:???? self.finalize_options() ??? stretch64:?? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 327, in finalize_options ??? stretch64:???? ep.load()(self, ep.name, value) ??? stretch64:?? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools_scm/integration.py", line 22, in version_keyword ??? stretch64:???? dist.metadata.version = get_version(**value) ??? stretch64:?? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 119, in get_version ??? stretch64:???? parsed_version = _do_parse(root, parse) ??? stretch64:?? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 97, in _do_parse ??? stretch64:???? "use git+https://github.com/user/proj.git#egg=proj" % root) ??? stretch64: LookupError: setuptools-scm was unable to detect version for '/vagrant/borg/borg'. ??? stretch64: ??? stretch64: Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work. ??? stretch64: ??? stretch64: For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj ??? stretch64: Obtaining file:///vagrant/borg/borg ??? stretch64:???? Complete output from command python setup.py egg_info: ??? stretch64:???? Detected and preferring liblz4 over bundled LZ4 ??? stretch64:???? Traceback (most recent call last): ??? stretch64:?????? File "", line 20, in ??? stretch64:?????? File "/vagrant/borg/borg/setup.py", line 856, in ??? stretch64:???????? extras_require=extras_require, ??? stretch64:?????? File "/usr/lib/python3.5/distutils/core.py", line 108, in setup ??? stretch64:???????? _setup_distribution = dist = klass(attrs) ??? stretch64:?????? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 272, in __init__ ??? stretch64:???????? _Distribution.__init__(self,attrs) ??? stretch64:?????? File "/usr/lib/python3.5/distutils/dist.py", line 281, in __init__ ??? stretch64:???????? self.finalize_options() ??? stretch64:?????? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 327, in finalize_options ??? stretch64:???????? ep.load()(self, ep.name, value) ??? stretch64:?????? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools_scm/integration.py", line 22, in version_keyword ??? stretch64:???????? dist.metadata.version = get_version(**value) ??? stretch64:?????? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 119, in get_version ??? stretch64:???????? parsed_version = _do_parse(root, parse) ??? stretch64:?????? File "/vagrant/borg/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 97, in _do_parse ??? stretch64:???????? "use git+https://github.com/user/proj.git#egg=proj" % root) ??? stretch64:???? LookupError: setuptools-scm was unable to detect version for '/vagrant/borg/borg'. ??? stretch64: ??? stretch64:???? Make sure you're either building from a fully intact git repository or PyPI tarballs. Most other sources (such as GitHub's tarballs, a git checkout without the .git folder) don't contain the necessary metadata and will not work. ??? stretch64: ??? stretch64:???? For example, if you're using pip, instead of https://github.com/user/proj/archive/master.zip use git+https://github.com/user/proj.git#egg=proj ??? stretch64: ??? stretch64:???? ---------------------------------------- <----- /SNIP stretch64 -----> From tw at waldmann-edv.de Wed Jun 20 09:25:50 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 20 Jun 2018 15:25:50 +0200 Subject: [Borgbackup] Build Static Borg Binary fail with Vagrantfile In-Reply-To: <8e607709-c918-3f33-ed63-bdc150b77ab2@12357.de> References: <8e607709-c918-3f33-ed63-bdc150b77ab2@12357.de> Message-ID: <389f6279-d3c1-1c98-9276-838207b6a03d@waldmann-edv.de> > i'd like to backup my Centos7 and Debian Stretch Systems with borgbackup. > Because the versions from the Repositorys are different and old, stretch-backports has 1.1.5 > i decided to build the v1.1.6 with the Vagrantfile as static binary, there are ready-to-use binaries, see "github releases". > but that's failed with the error below for both Vagrant boxes (centos7_64 & stretch64) the 1.1.x binaries were made on debian wheezy (for better compat with old linuxes) and they should work on any linux that has the required glibc. see Vagrantfile in 1.1-maint git branch. in the future, this might be moved to debian jessie as wheezy has run out of security support. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From lars-borgbackup at 12357.de Thu Jun 21 07:04:21 2018 From: lars-borgbackup at 12357.de (Lars Bendel) Date: Thu, 21 Jun 2018 13:04:21 +0200 Subject: [Borgbackup] Build Static Borg Binary fail with Vagrantfile In-Reply-To: <389f6279-d3c1-1c98-9276-838207b6a03d@waldmann-edv.de> References: <8e607709-c918-3f33-ed63-bdc150b77ab2@12357.de> <389f6279-d3c1-1c98-9276-838207b6a03d@waldmann-edv.de> Message-ID: Hi Thomas: >> i'd like to backup my Centos7 and Debian Stretch Systems with borgbackup. >> Because the versions from the Repositorys are different and old, > stretch-backports has 1.1.5 > >> i decided to build the v1.1.6 with the Vagrantfile as static binary, > there are ready-to-use binaries, see "github releases". the backports i haven't seen, but befor i mix Version 1.1.5 (on Debian) and 1.1.4 (on Centos) for a repository over ssh, i will fall back for the moment to the pre-compiled binary. > >> but that's failed with the error below for both Vagrant boxes (centos7_64 & stretch64) > the 1.1.x binaries were made on debian wheezy (for better compat with > old linuxes) and they should work on any linux that has the required glibc. > > see Vagrantfile in 1.1-maint git branch. > > in the future, this might be moved to debian jessie as wheezy has run > out of security support. Because the b.vm.box "debian7-amd64" did not exist anymore, i used the "debian/wheezy64" box and added to install the "curl" package to the vagrantfile. Then i have test the build with the wheezy64 box but get the same error with "vagrant up" for Centos7 and Stretch at the point "python setup.py clean" <---- SNIP ----> def install_borg(fuse) script = <<-EOF . ~/.bash_profile cd /vagrant/borg . borg-env/bin/activate pip install -U wheel # upgrade wheel, too old for 3.5 cd borg pip install -r requirements.d/development.txt => python setup.py clean <---- /SNIP ----> if i execute the commands direct in debug mode, it's looks like the borg version can't be parsed or found but i don't know why or what i have forgotten? Unfortunately C(++), PHP and BASH scripting is no problem but python i have never used. :-( <---- SNIP2 ---> (borg-env) vagrant at wheezy:/vagrant/borg/borg$ SETUPTOOLS_SCM_DEBUG=1 python setup.py clean looking for ep setuptools_scm.parse_scm_fallback . root '/vagrant/borg/borg' looking for ep setuptools_scm.parse_scm /vagrant/borg/borg looking for ep setuptools_scm.parse_scm_fallback /vagrant/borg/borg Traceback (most recent call last): File "setup.py", line 856, in extras_require=extras_require, File "/home/vagrant/.pyenv/versions/3.5.5/lib/python3.5/distutils/core.py", line 108, in setup _setup_distribution = dist = klass(attrs) File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 318, in __init__ _Distribution.__init__(self, attrs) File "/home/vagrant/.pyenv/versions/3.5.5/lib/python3.5/distutils/dist.py", line 281, in __init__ self.finalize_options() File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools/dist.py", line 376, in finalize_options ep.load()(self, ep.name, value) File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/integration.py", line 22, in version_keyword dist.metadata.version = get_version(**value) File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 119, in get_version parsed_version = _do_parse(root, parse) File "/home/vagrant/.pyenv/versions/3.5.5/envs/borg-env/lib/python3.5/site-packages/setuptools_scm/__init__.py", line 97, in _do_parse "use git+https://github.com/user/proj.git#egg=proj" % root) LookupError: setuptools-scm was unable to detect version for '/vagrant/borg/borg'. <---- /SNIP2 ---> best regards ?? Lars Bendel From dave at gasaway.org Wed Jun 27 01:22:17 2018 From: dave at gasaway.org (David Gasaway) Date: Tue, 26 Jun 2018 22:22:17 -0700 Subject: [Borgbackup] Recovery from missing segments Message-ID: Hi, I'd like to ask if borg can recover gracefully from missing segments. That is, say a new segment is created and recorded, but the next time that borg runs, the segment file is no longer there. Will borg recognize this on the next backup and recreate the file? Or will it only see that the chunks were previously saved and move on? I'm not particularly concerned that a particular archive may be "corrupt", only that subsequent archives are intact. The docs for 'borg check' say about an archive check: "For all files (items) in the archive, for all chunks referenced by these files, check if chunk is present." Does that mean checking for existence of the segment where the chuck is expected to be stored, for the chunk object in the archive metadata, or both? I don't mind running a archive check, but I'm not sure catches the above scenario. In fact, I did run one, but it seemed to finish quickly without reporting any issue. Running a repository check is very painful, as in my configuration, it means downloading every segment. Thanks. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org From tw at waldmann-edv.de Wed Jun 27 07:05:55 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Jun 2018 13:05:55 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: References: Message-ID: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> > I'd like to ask if borg can recover gracefully from missing segments. > That is, say a new segment is created and recorded, but the next time > that borg runs, the segment file is no longer there. Will borg > recognize this on the next backup borg uses the client-side chunks index to determine whether a chunk "is present in the repo". the problem in the case pictured above is that your index does not correspond to the (corrupted) repo any more. to detect such corruption, there is borg check. also, it is a good idea to have multiple repositories on different targets (preferably at different places) so that in case one repo suffers from corruption (e.g. caused by bad hardware) or other issues, you still have another good repo. > The docs for 'borg check' say about an archive check: "For all files > (items) in the archive, for all chunks referenced by these files, > check if chunk is present." Does that mean checking for existence of > the segment where the chuck is expected to be stored, for the chunk > object in the archive metadata, or both? Without --verify-data, the archives check (2nd phase) will check the presence of the chunk (in the chunks index). The index will be a freshly built one from the repo check phase (first phase). It is built from the segment files, so if segment files are missing, chunks will be missing in the freshly built index, too. With --verify-data, it will also read and cryptographically verify each chunk (slow). In general, be aware that borg check is complex code, might be imperfect and can not do wonders anyway. If you lose segment files you very likely have lost data. Due to the deduplication, every chunk of data in a repo is present only once and if you lose that, you have lost that piece of data(*). Same thing applies to metadata. (*) you might be lucky if you still have the input data that produces this chunk. Files having lost that chunk might get healed by running borg check --repair, borg create, borg check --repair sequence. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From felix.schwarz at oss.schwarz.eu Wed Jun 27 07:16:25 2018 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Wed, 27 Jun 2018 13:16:25 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> Message-ID: Am 27.06.2018 um 13:05 schrieb Thomas Waldmann: > also, it is a good idea to have multiple repositories on different > targets (preferably at different places) so that in case one repo > suffers from corruption (e.g. caused by bad hardware) or other issues, > you still have another good repo. Sorry to "hijack" the thread but I have a follow-up question here: So let's say I have a repo on an external disk. Now I buy a second disk and just copy all the files from the first disk. Is there anything I need to do before using either disk as a backup target in the future? (e.g. as both repos are "identical" and might contain the same "unique" ID the client cache might get confused) I'd like to avoid recreating a multi-TB repo from scratch. Thanks, Felix From tw at waldmann-edv.de Wed Jun 27 08:23:49 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Jun 2018 14:23:49 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> Message-ID: <0ce62d6b-95d7-608a-c608-dd0a5c94d7a7@waldmann-edv.de> > So let's say I have a repo on an external disk. Now I buy a second disk and ... then you do a "borg init" on that. > just copy all the files from the first disk. no, don't do that. this would lead to multiple problems: - crypto unsecure due to AES CTR reuse - duplicate repo id and thus cache confusion > I'd like to avoid recreating a multi-TB repo from scratch. that's the only really easy and safe way. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From public at enkore.de Wed Jun 27 09:24:10 2018 From: public at enkore.de (Marian Beermann) Date: Wed, 27 Jun 2018 15:24:10 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> Message-ID: <9d953a4b-9646-4f5f-7bd0-18f4e16b947f@enkore.de> If it's encrypted there is no way to do this. *Always remember:* Borg encryption is fundamentally broken as soon as - you have multiple copies of the same repo (this includes restoring a repo from backup, if you do that sort of thing), - or multiple clients accessing the same untrusted server. If it's not encrypted you want to change the repo ID in the repo config file. Cheers, Marian On 27.06.2018 13:16, Felix Schwarz wrote: > > Am 27.06.2018 um 13:05 schrieb Thomas Waldmann: >> also, it is a good idea to have multiple repositories on different >> targets (preferably at different places) so that in case one repo >> suffers from corruption (e.g. caused by bad hardware) or other issues, >> you still have another good repo. > > Sorry to "hijack" the thread but I have a follow-up question here: > So let's say I have a repo on an external disk. Now I buy a second disk and > just copy all the files from the first disk. > > Is there anything I need to do before using either disk as a backup target in > the future? (e.g. as both repos are "identical" and might contain the same > "unique" ID the client cache might get confused) > > I'd like to avoid recreating a multi-TB repo from scratch. > > Thanks, > Felix > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From mhossbach at gmail.com Wed Jun 27 09:35:03 2018 From: mhossbach at gmail.com (=?UTF-8?Q?Martin_Ho=C3=9Fbach?=) Date: Wed, 27 Jun 2018 15:35:03 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: <9d953a4b-9646-4f5f-7bd0-18f4e16b947f@enkore.de> References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> <9d953a4b-9646-4f5f-7bd0-18f4e16b947f@enkore.de> Message-ID: Hi On Wed, Jun 27, 2018 at 3:24 PM, Marian Beermann wrote: > Borg encryption is fundamentally broken as soon as > - you have multiple copies of the same repo (this includes restoring a > repo from backup, if you do that sort of thing), > I'm sorry, but why is that the case? Martin -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Wed Jun 27 09:40:18 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Jun 2018 15:40:18 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: <9d953a4b-9646-4f5f-7bd0-18f4e16b947f@enkore.de> References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> <9d953a4b-9646-4f5f-7bd0-18f4e16b947f@enkore.de> Message-ID: <1ca22f93-ca12-ee92-171c-6802cb97b2d1@waldmann-edv.de> > Borg encryption is fundamentally broken as soon as > - you have multiple copies of the same repo (this includes restoring a > repo from backup, if you do that sort of thing), just to be clear: "broken" means "unsecure against attackers" here. not "broken, so you can't decrypt it any more". still nothing you'ld want to have, so do not copy encrypted repos and independently update original and copy. > - or multiple clients accessing the same untrusted server. guess you meant "... the same repository on an untrusted server". -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Wed Jun 27 09:42:27 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Jun 2018 15:42:27 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: <1ca22f93-ca12-ee92-171c-6802cb97b2d1@waldmann-edv.de> References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> <9d953a4b-9646-4f5f-7bd0-18f4e16b947f@enkore.de> <1ca22f93-ca12-ee92-171c-6802cb97b2d1@waldmann-edv.de> Message-ID: <42e668af-5419-3475-8e17-1a6db87023bf@waldmann-edv.de> I opened this issue to check/update the docs about "repo copies": https://github.com/borgbackup/borg/issues/3933 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From dave at gasaway.org Wed Jun 27 11:50:17 2018 From: dave at gasaway.org (David Gasaway) Date: Wed, 27 Jun 2018 08:50:17 -0700 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> Message-ID: On Wed, Jun 27, 2018 at 8:38 AM, David Gasaway wrote: > Right, but again, I'm talking about the loss of new segment files and > I don't care that a particular archive is corrupt. Also, if there is some sequence (like delete then prune) that would effectively remove the new chunks from the chunk index so they are recreated on the next backup, I wouldn't mind using that. Would that work? Thanks. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org From dave at gasaway.org Wed Jun 27 11:38:50 2018 From: dave at gasaway.org (David Gasaway) Date: Wed, 27 Jun 2018 08:38:50 -0700 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> Message-ID: On Wed, Jun 27, 2018 at 4:05 AM, Thomas Waldmann wrote: > borg uses the client-side chunks index to determine whether a chunk "is > present in the repo". the problem in the case pictured above is that > your index does not correspond to the (corrupted) repo any more. This much I understand. > to detect such corruption, there is borg check. Yes, but I'm trying to find a check mode that will repair this condition without reading the segment files, as that would take a very long time. > also, it is a good idea to have multiple repositories on different > targets (preferably at different places) so that in case one repo > suffers from corruption (e.g. caused by bad hardware) or other issues, > you still have another good repo. I do have a second repo, but that doesn't help me repair the first. > Without --verify-data, the archives check (2nd phase) will check the > presence of the chunk (in the chunks index). The index will be a freshly > built one from the repo check phase (first phase). It is built from the > segment files, so if segment files are missing, chunks will be missing > in the freshly built index, too. I'm trying to avoid a repo check as it downloads all the segment files. I believe you are saying that '--archives-only' only checks the index and would do nothing to detect or repair the condition I described. Can such a mode be added? Hypothetical mode would simply check the filesystem for the expected segment files, and if not found, replace the associated chunks in the index with zero chunks so the chunks/segments get recreated at the next backup (assuming source data is still present). Again, I don't care that a particular archive is corrupt or has references to missing chucks, so long as I can get the backups back on track. > If you lose segment files you very likely have lost data. Due to the > deduplication, every chunk of data in a repo is present only once and if > you lose that, you have lost that piece of data(*). Same thing applies > to metadata. Right, but again, I'm talking about the loss of new segment files and I don't care that a particular archive is corrupt. > (*) you might be lucky if you still have the input data that produces > this chunk. Files having lost that chunk might get healed by running > borg check --repair, borg create, borg check --repair sequence. My source data isn't going anywhere. Or if it has, it's no great loss so long as I have the most recent data. Thanks. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org From felix.schwarz at oss.schwarz.eu Wed Jun 27 14:19:40 2018 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Wed, 27 Jun 2018 20:19:40 +0200 Subject: [Borgbackup] Recovery from missing segments In-Reply-To: <0ce62d6b-95d7-608a-c608-dd0a5c94d7a7@waldmann-edv.de> References: <4251872c-73d3-895e-c02d-61289d4d8179@waldmann-edv.de> <0ce62d6b-95d7-608a-c608-dd0a5c94d7a7@waldmann-edv.de> Message-ID: <9a902c8a-18a4-6215-aaf4-a2af48381dc2@oss.schwarz.eu> Am 27.06.2018 um 14:23 schrieb Thomas Waldmann: > no, don't do that. > > this would lead to multiple problems: > - crypto unsecure due to AES CTR reuse > - duplicate repo id and thus cache confusion > >> I'd like to avoid recreating a multi-TB repo from scratch. > > that's the only really easy and safe way. Thank you for the advice - seems like I dodged a bullet here... Felix