From tw at waldmann-edv.de Mon Oct 1 04:42:09 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 1 Oct 2018 10:42:09 +0200 Subject: [Borgbackup] questions about append-only mode repository In-Reply-To: References: Message-ID: The very fundamental problem we are running into here is: "making sure the backups you have are all you need" There are currently 2 solutions to this: a) never switch off append-only (will use increasing amounts of space, might run into scalability issues for huge archive counts and some operations which are O(N_archives)) this of course assumes that you didn't make bad mistakes when creating your backups, like forgetting some important stuff or backing up inconsistent data or ... b) make sure the backups you have are all you need - before leaving append-only mode / actually deleting stuff borg does not support you with b) except offering borg list repo/archive, borg check and maybe some other basic operations. you can see that as a missing feature or as just being out of scope. I don't see append-only as negative as Marian (who seems to be rather negative about borg in general since he stopped contributing a while ago - after doing a lot in the past). It needs to be used carefully, but we have some docs that describe how to use append-only mode in case something bad happened. If these are not sufficient: PRs welcome. Ranting about append-only is not really helping as long as we have no better alternative. ---- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From abelschreck3 at freenet.de Thu Oct 4 08:44:46 2018 From: abelschreck3 at freenet.de (Christian) Date: Thu, 4 Oct 2018 14:44:46 +0200 Subject: [Borgbackup] Can backups be combined? Message-ID: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> Hi altogether, does anybody know of a method to combine 2 or several backups? Example: I have a folder on my system which is called "python-course". In it I have a video-file called "lesson_1.mp4". I want to backup that folder. So I use the command? "borg create -p -s -v -C lz4 /media/rosika/A492-CD29/borgbackups::python-course_backup /home/rosika/Dokumente/python-course". Next step: I delete "lesson_1.mp4" in the original folder in order to save space and instead store "lesson_2.mp4" in the folder. To backup that one I use the command "borg create -p -s -v -C lz4 /media/rosika/A492-CD29/borgbackups::2python-course_backup /home/rosika/Dokumente/python-course". And so on. Finally I have n backups containing one lesson per backup. But what if I want to playback all of those lessons/backups or even mount them? Is there a way of combining the backups "lesson_1.mp4" to "lesson_n.mp4"? Thanks a lot for your help in advance. Gretings Rosika P.S.: system: Linux/Lubuntu 16.04.5 LTS, 64 bit From abelschreck3 at freenet.de Thu Oct 4 09:48:46 2018 From: abelschreck3 at freenet.de (Christian) Date: Thu, 4 Oct 2018 15:48:46 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> Message-ID: Hi Gerrit, thanks a lot for your quick answer. Yeah, I?m? aware of the fact that it?s not a backup task. If I understand you correctly in my scenario I?d have to mount, say three archives, like this: 1. ) borg mount /media/rosika/A492-CD29/borgbackups::python-course_backup /home/rosika/Dokumente/wiederhergestellt 2.)? borg mount /media/rosika/A492-CD29/borgbackups::2python-course_backup /home/rosika/Dokumente/wiederhergestellt 3.)? borg mount /media/rosika/A492-CD29/borgbackups::3python-course_backup /home/rosika/Dokumente/wiederhergestellt So they?re available inside the same path at the same time. I hope that?s correct that way. Just another question: Am I right in supposing that when deleting e.g. the first backup the content of that one will be lost? So I?d have to keep all of them for my purposes. Greetings Rosika Am 04.10.2018 um 15:29 schrieb Gerrit A. Smit: > > Hello, > > > That is not a backup task. > > But you can mount several archives using 'borg mount ...' and, by > doing so, have them available inside one path. > > > Gerrit > From gait at atcomputing.nl Thu Oct 4 09:57:06 2018 From: gait at atcomputing.nl (Gerrit A. Smit) Date: Thu, 4 Oct 2018 15:57:06 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> Message-ID: <631ceb8b-1fec-3998-e787-67d52a1adbdb@atcomputing.nl> Christian schreef op 04-10-18 om 15:48: > Hi Gerrit, > > thanks a lot for your quick answer. > > Yeah, I?m? aware of the fact that it?s not a backup task. > > If I understand you correctly in my scenario I?d have to mount, say > three archives, like this: Please, do not use the same mount point multiple times to mount on! That would be pretty useless, as only the last mount will work. What I meant is using these mount points whg/1p/ whg/2p/ whg/3p/ so alle Folgen are inside whg/ > 1. ) borg mount > /media/rosika/A492-CD29/borgbackups::python-course_backup > /home/rosika/Dokumente/wiederhergestellt > 2.)? borg mount > /media/rosika/A492-CD29/borgbackups::2python-course_backup > /home/rosika/Dokumente/wiederhergestellt > 3.)? borg mount > /media/rosika/A492-CD29/borgbackups::3python-course_backup > /home/rosika/Dokumente/wiederhergestellt > > So they?re available inside the same path at the same time. > I hope that?s correct that way. > > Just another question: > Am I right in supposing that when deleting e.g. the first backup the > content of that one will be lost? So I?d have to keep all of them for my > purposes. Yes, I think so. Gerrit -------------- next part -------------- An HTML attachment was scrubbed... URL: From gait at atcomputing.nl Thu Oct 4 09:31:28 2018 From: gait at atcomputing.nl (Gerrit A. Smit) Date: Thu, 4 Oct 2018 15:31:28 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> Message-ID: <511197e4-01ae-147c-4b90-612e099b4630@atcomputing.nl> Hello, That is not a backup task. But you can mount several archives using 'borg mount ...' and, by doing so, have them available inside one path. Gerrit -------------- next part -------------- An HTML attachment was scrubbed... URL: From abelschreck3 at freenet.de Thu Oct 4 10:15:19 2018 From: abelschreck3 at freenet.de (Christian) Date: Thu, 4 Oct 2018 16:15:19 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: <631ceb8b-1fec-3998-e787-67d52a1adbdb@atcomputing.nl> References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> <631ceb8b-1fec-3998-e787-67d52a1adbdb@atcomputing.nl> Message-ID: <7f272c54-a73c-b1f1-bfe0-cba011493e28@freenet.de> Hi Gerrit, thanks again. O.K., now I understand. So inside my "wiederhergestellt"-folder the sub-folders 1p, 2p and 3p are created. Within each of them one backup. You helped me a lot in using borgbackup in a more efficient way. Thanks again and have a nice day. Greetings Rosika Am 04.10.2018 um 15:57 schrieb Gerrit A. Smit: > > > Christian schreef op 04-10-18 om 15:48: >> Hi Gerrit, >> >> thanks a lot for your quick answer. >> >> Yeah, I?m? aware of the fact that it?s not a backup task. >> >> If I understand you correctly in my scenario I?d have to mount, say >> three archives, like this: > > Please, do not use the same mount point multiple times to mount on! > > That would be pretty useless, as only the last mount will work. > > What I meant is using these mount points > > whg/1p/ > > whg/2p/ > > whg/3p/ > > so alle Folgen are inside whg/ > >> 1. ) borg mount >> /media/rosika/A492-CD29/borgbackups::python-course_backup >> /home/rosika/Dokumente/wiederhergestellt >> 2.)? borg mount >> /media/rosika/A492-CD29/borgbackups::2python-course_backup >> /home/rosika/Dokumente/wiederhergestellt >> 3.)? borg mount >> /media/rosika/A492-CD29/borgbackups::3python-course_backup >> /home/rosika/Dokumente/wiederhergestellt >> >> So they?re available inside the same path at the same time. >> I hope that?s correct that way. >> >> Just another question: >> Am I right in supposing that when deleting e.g. the first backup the >> content of that one will be lost? So I?d have to keep all of them for my >> purposes. > > Yes, I think so. > > > Gerrit > From abelschreck3 at freenet.de Thu Oct 4 10:36:28 2018 From: abelschreck3 at freenet.de (Christian) Date: Thu, 4 Oct 2018 16:36:28 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: <47dfc669-8b65-8954-4b54-ed187a96eb48@atcomputing.nl> References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> <631ceb8b-1fec-3998-e787-67d52a1adbdb@atcomputing.nl> <7f272c54-a73c-b1f1-bfe0-cba011493e28@freenet.de> <47dfc669-8b65-8954-4b54-ed187a96eb48@atcomputing.nl> Message-ID: <71301db1-2742-2e5a-0e7a-661913233a50@freenet.de> Hi again, thanks for the hint. I was too quick in trying it out. And indeed I didn?t create those sub-folders beforehand. And of course I got the message: "/home/rosika/Dokumente/wiederhergestellt/1p: Mountpoint must be a writable directory". O.K., I thought, and then created three sub-folders as per your instruction. Then it worked. So everything works fine now. Tnx again. Rosika Am 04.10.2018 um 16:27 schrieb Gerrit A. Smit: > Christian schreef op 04-10-18 om 16:15: >> O.K., now I understand. >> So inside my "wiederhergestellt"-folder the sub-folders 1p, 2p and 3p >> are created. Within each of them one backup. > > Well, just to be sure: you should create those directories yourself > before they can be used as mount points (by borg or whatever)! > > > Gerrit > From tw at waldmann-edv.de Thu Oct 4 10:54:05 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 4 Oct 2018 16:54:05 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: <71301db1-2742-2e5a-0e7a-661913233a50@freenet.de> References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> <631ceb8b-1fec-3998-e787-67d52a1adbdb@atcomputing.nl> <7f272c54-a73c-b1f1-bfe0-cba011493e28@freenet.de> <47dfc669-8b65-8954-4b54-ed187a96eb48@atcomputing.nl> <71301db1-2742-2e5a-0e7a-661913233a50@freenet.de> Message-ID: maybe the most convenient way is to use borg mount with the "version(s?)" mount-option. that will produce a merged view of all archives selected / of all dirs/files selected. you can only mount SOME archives (not all), e.g. the first 3 or last 5 or matching some archive name pattern, see the "borg mount" options.. you can only make SOME files/dirs in these archives visible below the mountpoint (not all), by giving the (relative) full base path. this is useful if you already know the precise base path and if there is also a lot of other stuff in the archives (which you do not want to use in that case). please see the docs for details and just try it. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From gait at atcomputing.nl Thu Oct 4 10:27:10 2018 From: gait at atcomputing.nl (Gerrit A. Smit) Date: Thu, 4 Oct 2018 16:27:10 +0200 Subject: [Borgbackup] Can backups be combined? In-Reply-To: <7f272c54-a73c-b1f1-bfe0-cba011493e28@freenet.de> References: <3999438e-312f-78d4-d196-e2472db54b3f@freenet.de> <631ceb8b-1fec-3998-e787-67d52a1adbdb@atcomputing.nl> <7f272c54-a73c-b1f1-bfe0-cba011493e28@freenet.de> Message-ID: <47dfc669-8b65-8954-4b54-ed187a96eb48@atcomputing.nl> Christian schreef op 04-10-18 om 16:15: > O.K., now I understand. > So inside my "wiederhergestellt"-folder the sub-folders 1p, 2p and 3p > are created. Within each of them one backup. Well, just to be sure: you should create those directories yourself before they can be used as mount points (by borg or whatever)! Gerrit -------------- next part -------------- An HTML attachment was scrubbed... URL: From abelschreck3 at freenet.de Fri Oct 5 06:50:26 2018 From: abelschreck3 at freenet.de (Christian) Date: Fri, 5 Oct 2018 12:50:26 +0200 Subject: [Borgbackup] Can backups be combined? Message-ID: <672c875a-3be2-947a-5234-c0f268d349cc@freenet.de> Hello Thomas, thank you so much for your reply. I tried out what you suggested and it works very well. The command "borg mount -o versions /media/rosika/A492-CD29/borgbackups /home/rosika/Dokumente/wiederhergestellt/1p" indeed merges all the backups of the repository. That?s exactly what I was looking for. Tnx again and many greetings. Rosika From tw at waldmann-edv.de Sat Oct 6 14:35:20 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 6 Oct 2018 20:35:20 +0200 Subject: [Borgbackup] borg quotas not enforced due to a bug Message-ID: <07d55e98-9ac2-5294-1dac-b8ceb7a79cab@waldmann-edv.de> PSA: if you run borg 1.1.x (x<8) as a service / storage provider for 3rd parties and you rely on borg's quota feature, you likely want to patch borg until this will be fixed in a future 1.1.8 release: https://github.com/borgbackup/borg/issues/4093 Package maintainers please also should patch. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tarjei at asku.no Fri Oct 12 02:40:59 2018 From: tarjei at asku.no (Tarjei Huse) Date: Fri, 12 Oct 2018 08:40:59 +0200 Subject: [Borgbackup] A question wrt to backupstrategies Message-ID: Hi, I got a question with regard to backup strategies. I use borg for backing up 4-5 ubuntu boxen running fairly standard LAMP applications. What I'm wondering about is which strategy I should use for backups. Right now I use a separate archive path for each box. Is this a good strategy or should I merge all the archives into one with separate paths within the archive for each box (is that even possible?). Also, I want to take backups very often of some files, mainly the ones that have been uploaded by users. Can I run backups for just some paths against the same archive without loosing other data during pruning etc? Kind regards Tarjei -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at gasaway.org Fri Oct 12 03:16:38 2018 From: dave at gasaway.org (David Gasaway) Date: Fri, 12 Oct 2018 00:16:38 -0700 Subject: [Borgbackup] A question wrt to backupstrategies In-Reply-To: References: Message-ID: On Thu, Oct 11, 2018 at 11:41 PM Tarjei Huse wrote: > Hi, > > I got a question with regard to backup strategies. > > I use borg for backing up 4-5 ubuntu boxen running fairly standard LAMP > applications. > > What I'm wondering about is which strategy I should use for backups. Right > now I use a separate archive path for each box. Is this a good strategy or > should I merge all the archives into one with separate paths within the > archive for each box (is that even possible?). > No. When creating a backup, you specify an archive name and source paths; there is no "path within the archive" so to speak. What you can do is put the machine name in the archive name and use '--prefix' on your prunes. This allows deduplication across sources. This is all explained in detail in the docs. Personally, I use separate repositories per source machine. > Also, I want to take backups very often of some files, mainly the ones > that have been uploaded by users. Can I run backups for just some paths > against the same archive without loosing other data during pruning etc? > Data is removed when every copy has been pruned. Let's say you have two backup jobs targeting a repository sourced from different paths, A and B. After a while, you have A, B, A', B', A'', B'', etc.. As long as your pruning strategy does not eliminate all sets of A (i.e., one that left only B''), then you still have A. You could also make use of archive names and --prefix as above to define separate pruning strategies for each data set. Heck, if you design your strategy appropriately, you could have both the machine name and data set name in the archive name, multiple sets for each machine, and define different pruning strategies for each subset. -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Tue Oct 16 03:13:54 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Tue, 16 Oct 2018 15:13:54 +0800 Subject: [Borgbackup] Multi-machine backup when there's some data overlap Message-ID: Apologies in advance if I get terminology wrong, I've used rsnapshot for a decade or more and am only now looking to upgrade my backup process. Having said that, what's the best way to approach multi-machine backup? Obviously the most straightforward way is to initialize at least one borg backup per machine. However if I read that correctly this will create isolated repositories, which cannot take advantage of any data overlap. Just to avoid this being an XY problem, my core use case is having multiple computers (Linux and Windows via cygwin) backing up to a single backup host using borgbackup. Some of these machines have duplicated data (especially in terms of photos/videos, since these are personal/family machines), so there should be significant space savings from being keeping them in one repository, if that is at all possible. Thanks in advace! -------------- next part -------------- An HTML attachment was scrubbed... URL: From panayotis at panayotis.com Tue Oct 16 08:49:09 2018 From: panayotis at panayotis.com (Panayotis Katsaloulis) Date: Tue, 16 Oct 2018 15:49:09 +0300 Subject: [Borgbackup] Multi-machine backup when there's some data overlap In-Reply-To: References: Message-ID: I personally backup more than one computer which have mostly common files for a long time without any problem. The only "issue" is that every machine before backing up it needs to fetch the other machines indices. It takes some time but it is acceptable. ???? ???, 16 ??? 2018, 3:36 ?.?. ? ??????? Panayotis ??????: > I personally backup more than one computer which have mostly common files > for a long time without any problem. > > The only "issue" is that every maching before backing up it needs to fetch > the other machines indices. It takes some time but it is acceptable. > > ???? ???, 16 ??? 2018, 10:14 ?.?. ? ??????? Oon-Ee Ng < > ngoonee.talk at gmail.com> ??????: > >> Apologies in advance if I get terminology wrong, I've used rsnapshot for >> a decade or more and am only now looking to upgrade my backup process. >> >> Having said that, what's the best way to approach multi-machine backup? >> Obviously the most straightforward way is to initialize at least one borg >> backup per machine. However if I read that correctly this will create >> isolated repositories, which cannot take advantage of any data overlap. >> >> Just to avoid this being an XY problem, my core use case is having >> multiple computers (Linux and Windows via cygwin) backing up to a single >> backup host using borgbackup. Some of these machines have duplicated data >> (especially in terms of photos/videos, since these are personal/family >> machines), so there should be significant space savings from being keeping >> them in one repository, if that is at all possible. >> >> Thanks in advace! >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clickwir at gmail.com Tue Oct 16 09:18:01 2018 From: clickwir at gmail.com (Zack Coffey) Date: Tue, 16 Oct 2018 07:18:01 -0600 Subject: [Borgbackup] Multi-machine backup when there's some data overlap In-Reply-To: References: Message-ID: https://borgbackup.readthedocs.io/en/stable/faq.html That question is the very first answer in the FAQ. The answer is basically "yes, but..." Each machine is responsible for taking care of the whole repo as if it were it's own. So it needs to sync the whole thing, take care of everything and leave it all in a good state. Otherwise the next backup client is going to have trouble. Oh, and only 1 can run at a time. If all of those are not a concern and dedupe is a higher priority, then yes they can all backup to 1 repo. Otherwise, individual repos is wiser. On Tue, Oct 16, 2018 at 1:14 AM Oon-Ee Ng wrote: > Apologies in advance if I get terminology wrong, I've used rsnapshot for a > decade or more and am only now looking to upgrade my backup process. > > Having said that, what's the best way to approach multi-machine backup? > Obviously the most straightforward way is to initialize at least one borg > backup per machine. However if I read that correctly this will create > isolated repositories, which cannot take advantage of any data overlap. > > Just to avoid this being an XY problem, my core use case is having > multiple computers (Linux and Windows via cygwin) backing up to a single > backup host using borgbackup. Some of these machines have duplicated data > (especially in terms of photos/videos, since these are personal/family > machines), so there should be significant space savings from being keeping > them in one repository, if that is at all possible. > > Thanks in advace! > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Tue Oct 16 11:21:31 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Tue, 16 Oct 2018 23:21:31 +0800 Subject: [Borgbackup] Multi-machine backup when there's some data overlap In-Reply-To: References: Message-ID: I read that, and figured it meant "no you can't do that directly" since having to rebuild indices seems like it would make backups take significantly longer. Would there be a clever way of sharing the indice/cache which would help with that? On Tue, 16 Oct 2018, 9:18 pm Zack Coffey, wrote: > https://borgbackup.readthedocs.io/en/stable/faq.html > > That question is the very first answer in the FAQ. > > The answer is basically "yes, but..." > > Each machine is responsible for taking care of the whole repo as if it > were it's own. So it needs to sync the whole thing, take care of everything > and leave it all in a good state. Otherwise the next backup client is going > to have trouble. Oh, and only 1 can run at a time. > > If all of those are not a concern and dedupe is a higher priority, then > yes they can all backup to 1 repo. Otherwise, individual repos is wiser. > > On Tue, Oct 16, 2018 at 1:14 AM Oon-Ee Ng wrote: > >> Apologies in advance if I get terminology wrong, I've used rsnapshot for >> a decade or more and am only now looking to upgrade my backup process. >> >> Having said that, what's the best way to approach multi-machine backup? >> Obviously the most straightforward way is to initialize at least one borg >> backup per machine. However if I read that correctly this will create >> isolated repositories, which cannot take advantage of any data overlap. >> >> Just to avoid this being an XY problem, my core use case is having >> multiple computers (Linux and Windows via cygwin) backing up to a single >> backup host using borgbackup. Some of these machines have duplicated data >> (especially in terms of photos/videos, since these are personal/family >> machines), so there should be significant space savings from being keeping >> them in one repository, if that is at all possible. >> >> Thanks in advace! >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at enkore.de Tue Oct 16 11:40:40 2018 From: public at enkore.de (Marian Beermann) Date: Tue, 16 Oct 2018 17:40:40 +0200 Subject: [Borgbackup] Multi-machine backup when there's some data overlap In-Reply-To: References: Message-ID: <643d7b72-9b57-0f92-3f0f-8af55d91c8da@enkore.de> And if you are using encryption you kinda need to trust the repo server as well. See https://borgbackup.readthedocs.io/en/stable/internals/security.html#attack-model On 10/16/18 3:18 PM, Zack Coffey wrote: > https://borgbackup.readthedocs.io/en/stable/faq.html > > That question is the very first answer in the FAQ. > > The answer is basically "yes, but..." > > Each machine is responsible for taking care of the whole repo as if it > were it's own. So it needs to sync the whole thing, take care of > everything and leave it all in a good state. Otherwise the next backup > client is going to have trouble. Oh, and only 1 can run at a time. > > If all of those are not a concern and dedupe is a higher priority, then > yes they can all backup to 1 repo. Otherwise, individual repos is wiser. > > On Tue, Oct 16, 2018 at 1:14 AM Oon-Ee Ng > wrote: > > Apologies in advance if I get terminology wrong, I've used rsnapshot > for a decade or more and am only now looking to upgrade my backup > process. > > Having said that, what's the best way to approach multi-machine > backup? Obviously the most straightforward way is to initialize at > least one borg backup per machine. However if I read that correctly > this will create isolated repositories, which cannot take advantage > of any data overlap. > > Just to avoid this being an XY problem, my core use case is having > multiple computers (Linux and Windows via cygwin) backing up to a > single backup host using borgbackup. Some of these machines have > duplicated data (especially in terms of photos/videos, since these > are personal/family machines), so there should be significant space > savings from being keeping them in one repository, if that is at all > possible. > > Thanks in advace! > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From ngoonee.talk at gmail.com Tue Oct 16 18:38:06 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Wed, 17 Oct 2018 06:38:06 +0800 Subject: [Borgbackup] Multi-machine backup when there's some data overlap In-Reply-To: <643d7b72-9b57-0f92-3f0f-8af55d91c8da@enkore.de> References: <643d7b72-9b57-0f92-3f0f-8af55d91c8da@enkore.de> Message-ID: Thanks for that, but the loss of 3 and 4 is *probably* acceptable for this situation, as my backup location is a machine I control. On Tue, Oct 16, 2018 at 11:40 PM Marian Beermann wrote: > And if you are using encryption you kinda need to trust the repo server > as well. > > See > > https://borgbackup.readthedocs.io/en/stable/internals/security.html#attack-model > > On 10/16/18 3:18 PM, Zack Coffey wrote: > > https://borgbackup.readthedocs.io/en/stable/faq.html > > > > That question is the very first answer in the FAQ. > > > > The answer is basically "yes, but..." > > > > Each machine is responsible for taking care of the whole repo as if it > > were it's own. So it needs to sync the whole thing, take care of > > everything and leave it all in a good state. Otherwise the next backup > > client is going to have trouble. Oh, and only 1 can run at a time. > > > > If all of those are not a concern and dedupe is a higher priority, then > > yes they can all backup to 1 repo. Otherwise, individual repos is wiser. > > > > On Tue, Oct 16, 2018 at 1:14 AM Oon-Ee Ng > > wrote: > > > > Apologies in advance if I get terminology wrong, I've used rsnapshot > > for a decade or more and am only now looking to upgrade my backup > > process. > > > > Having said that, what's the best way to approach multi-machine > > backup? Obviously the most straightforward way is to initialize at > > least one borg backup per machine. However if I read that correctly > > this will create isolated repositories, which cannot take advantage > > of any data overlap. > > > > Just to avoid this being an XY problem, my core use case is having > > multiple computers (Linux and Windows via cygwin) backing up to a > > single backup host using borgbackup. Some of these machines have > > duplicated data (especially in terms of photos/videos, since these > > are personal/family machines), so there should be significant space > > savings from being keeping them in one repository, if that is at all > > possible. > > > > Thanks in advace! > > _______________________________________________ > > Borgbackup mailing list > > Borgbackup at python.org > > https://mail.python.org/mailman/listinfo/borgbackup > > > > > > _______________________________________________ > > Borgbackup mailing list > > Borgbackup at python.org > > https://mail.python.org/mailman/listinfo/borgbackup > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Thu Oct 25 17:18:26 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Fri, 26 Oct 2018 05:18:26 +0800 Subject: [Borgbackup] Resuming backup - does it require the same archive name? Message-ID: After some testing with my /etc backups, I modified the script to backup my ~ and left it overnight to run. I'm using a fairly standard archive name:- home-{hostname}-{now} Obviously the archive name depends on the date and time. If I cancel my overnight backup run later this morning, I understand there will be checkpoints for resuming. Does that require me to specify the archive name which I originally used? Or can I just do borg create (with the above name) again? -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at gasaway.org Thu Oct 25 23:02:29 2018 From: dave at gasaway.org (David Gasaway) Date: Thu, 25 Oct 2018 20:02:29 -0700 Subject: [Borgbackup] Resuming backup - does it require the same archive name? In-Reply-To: References: Message-ID: On Thu, Oct 25, 2018 at 2:22 PM Oon-Ee Ng wrote: > After some testing with my /etc backups, I modified the script to backup > my ~ and left it overnight to run. > > I'm using a fairly standard archive name:- home-{hostname}-{now} > > Obviously the archive name depends on the date and time. If I cancel my > overnight backup run later this morning, I understand there will be > checkpoints for resuming. Does that require me to specify the archive name > which I originally used? Or can I just do borg create (with the above name) > again? > >From the FAQ... https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there "You may use the same archive name as in previous attempt or a different one (e.g. if you always include the current datetime), it does not matter." -- -:-:- David K. Gasaway -:-:- Email: dave at gasaway.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Fri Oct 26 00:57:44 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Fri, 26 Oct 2018 12:57:44 +0800 Subject: [Borgbackup] Resuming backup - does it require the same archive name? In-Reply-To: References: Message-ID: Thank you! That's reassuring. On Fri, Oct 26, 2018 at 11:02 AM David Gasaway wrote: > > > On Thu, Oct 25, 2018 at 2:22 PM Oon-Ee Ng wrote: > >> After some testing with my /etc backups, I modified the script to backup >> my ~ and left it overnight to run. >> >> I'm using a fairly standard archive name:- home-{hostname}-{now} >> >> Obviously the archive name depends on the date and time. If I cancel my >> overnight backup run later this morning, I understand there will be >> checkpoints for resuming. Does that require me to specify the archive name >> which I originally used? Or can I just do borg create (with the above name) >> again? >> > > From the FAQ... > > https://borgbackup.readthedocs.io/en/stable/faq.html#if-a-backup-stops-mid-way-does-the-already-backed-up-data-stay-there > "You may use the same archive name as in previous attempt or a different > one (e.g. if you always include the current datetime), it does not matter." > > -- > -:-:- David K. Gasaway > -:-:- Email: dave at gasaway.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Sat Oct 27 11:01:07 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Sat, 27 Oct 2018 23:01:07 +0800 Subject: [Borgbackup] Multi-machine backup when there's some data overlap In-Reply-To: References: <643d7b72-9b57-0f92-3f0f-8af55d91c8da@enkore.de> Message-ID: Noticed another issue related to this, I'm trying to replicate my rsync setup where I ran some backups as root (/etc and similar) and others as my own user (home directory). Both to the same repo. This effectively meant I was trying to run a two-source backup, so lots of rebuilding of cache involved. As I understand it, this means I have a few choices:- 1. Accept the rebuilding time (it's kinda long for my data, so not likely) 2. Run everything as root 3. Have a separate repo for the user-backup The issue with 2 is that I can only run borg list/info etc to check on my data when logged in as root. This would be a partial issue with 3, since that wouldn't apply for my own-user-backup (which is what's more 'important' subjectively anyway). Is my understanding correct? On Wed, Oct 17, 2018 at 6:38 AM Oon-Ee Ng wrote: > Thanks for that, but the loss of 3 and 4 is *probably* acceptable for this > situation, as my backup location is a machine I control. > > On Tue, Oct 16, 2018 at 11:40 PM Marian Beermann wrote: > >> And if you are using encryption you kinda need to trust the repo server >> as well. >> >> See >> >> https://borgbackup.readthedocs.io/en/stable/internals/security.html#attack-model >> >> On 10/16/18 3:18 PM, Zack Coffey wrote: >> > https://borgbackup.readthedocs.io/en/stable/faq.html >> > >> > That question is the very first answer in the FAQ. >> > >> > The answer is basically "yes, but..." >> > >> > Each machine is responsible for taking care of the whole repo as if it >> > were it's own. So it needs to sync the whole thing, take care of >> > everything and leave it all in a good state. Otherwise the next backup >> > client is going to have trouble. Oh, and only 1 can run at a time. >> > >> > If all of those are not a concern and dedupe is a higher priority, then >> > yes they can all backup to 1 repo. Otherwise, individual repos is wiser. >> > >> > On Tue, Oct 16, 2018 at 1:14 AM Oon-Ee Ng > > > wrote: >> > >> > Apologies in advance if I get terminology wrong, I've used rsnapshot >> > for a decade or more and am only now looking to upgrade my backup >> > process. >> > >> > Having said that, what's the best way to approach multi-machine >> > backup? Obviously the most straightforward way is to initialize at >> > least one borg backup per machine. However if I read that correctly >> > this will create isolated repositories, which cannot take advantage >> > of any data overlap. >> > >> > Just to avoid this being an XY problem, my core use case is having >> > multiple computers (Linux and Windows via cygwin) backing up to a >> > single backup host using borgbackup. Some of these machines have >> > duplicated data (especially in terms of photos/videos, since these >> > are personal/family machines), so there should be significant space >> > savings from being keeping them in one repository, if that is at all >> > possible. >> > >> > Thanks in advace! >> > _______________________________________________ >> > Borgbackup mailing list >> > Borgbackup at python.org >> > https://mail.python.org/mailman/listinfo/borgbackup >> > >> > >> > _______________________________________________ >> > Borgbackup mailing list >> > Borgbackup at python.org >> > https://mail.python.org/mailman/listinfo/borgbackup >> > >> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Sat Oct 27 11:06:13 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Sat, 27 Oct 2018 23:06:13 +0800 Subject: [Borgbackup] Can rsynced repos be used independently after the initial rsync? Message-ID: The FAQ question on this says that syncing of the repo can be done using any tool (including rsync) as long as no backup is run to the repo while it's being synced. However it also notes that as a long-term solution this leaves much to be desired, since anything corrupting the first repo would be synced over and result in a corrupted second repo. The recommendation given is to keep backups separate (run borg create twice, once to each). Given I have a relatively large repo relative to my network connections throughput, and I want to have one borg repo on a local network backup machine and one on a remote backup machine, I was thinking about:- 1. Running rsync (would likely take slightly under a day) of the repo from local backup machine to remote. 2. Independently using both repos after rsync is successful. That means my backup script would call borg create repo1:/path followed by borg create repo2:/path Would this work, and does it have any downsides? -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at enkore.de Sat Oct 27 12:50:02 2018 From: public at enkore.de (Marian Beermann) Date: Sat, 27 Oct 2018 18:50:02 +0200 Subject: [Borgbackup] Can rsynced repos be used independently after the initial rsync? In-Reply-To: References: Message-ID: <202c8ffb-3f7f-58ca-98a8-5c8e622016fd@enkore.de> This can work, if you change the repository ID in the repository's config file in the two "forked" repos. Realize that all encryption / authentication offered by Borg is completely useless when used like this. Cheers, Marian Am 27.10.18 um 17:06 schrieb Oon-Ee Ng: > The FAQ question on this says that syncing of the repo can be done using > any tool (including rsync) as long as no backup is run to the repo while > it's being synced. > > However it also notes that as a long-term solution this leaves much to > be desired, since anything corrupting the first repo would be synced > over and result in a corrupted second repo. The recommendation given is > to keep backups separate (run borg create twice, once to each). > > Given I have a relatively large repo relative to my network connections > throughput, and I want to have one borg repo on a local network backup > machine and one on a remote backup machine, I was thinking about:- > > 1. Running rsync (would likely take slightly under a day) of the repo > from local backup machine to remote. > > 2. Independently using both repos after rsync is successful. That means > my backup script would call borg create repo1:/path followed by borg > create repo2:/path > > Would this work, and does it have any downsides? > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From ngoonee.talk at gmail.com Sat Oct 27 23:11:24 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Sun, 28 Oct 2018 11:11:24 +0800 Subject: [Borgbackup] Can rsynced repos be used independently after the initial rsync? In-Reply-To: <202c8ffb-3f7f-58ca-98a8-5c8e622016fd@enkore.de> References: <202c8ffb-3f7f-58ca-98a8-5c8e622016fd@enkore.de> Message-ID: Oh, how does that affect encryption or authentication? Is it just the fact that both repos use the exact same encryption key? Based on my (probably flawed) understanding that doesn't really invalidate security? On Sun, 28 Oct 2018, 12:50 am Marian Beermann, wrote: > This can work, if you change the repository ID in the repository's > config file in the two "forked" repos. > > Realize that all encryption / authentication offered by Borg is > completely useless when used like this. > > Cheers, Marian > > Am 27.10.18 um 17:06 schrieb Oon-Ee Ng: > > The FAQ question on this says that syncing of the repo can be done using > > any tool (including rsync) as long as no backup is run to the repo while > > it's being synced. > > > > However it also notes that as a long-term solution this leaves much to > > be desired, since anything corrupting the first repo would be synced > > over and result in a corrupted second repo. The recommendation given is > > to keep backups separate (run borg create twice, once to each). > > > > Given I have a relatively large repo relative to my network connections > > throughput, and I want to have one borg repo on a local network backup > > machine and one on a remote backup machine, I was thinking about:- > > > > 1. Running rsync (would likely take slightly under a day) of the repo > > from local backup machine to remote. > > > > 2. Independently using both repos after rsync is successful. That means > > my backup script would call borg create repo1:/path followed by borg > > create repo2:/path > > > > Would this work, and does it have any downsides? > > > > _______________________________________________ > > Borgbackup mailing list > > Borgbackup at python.org > > https://mail.python.org/mailman/listinfo/borgbackup > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From billk at iinet.net.au Sun Oct 28 08:04:39 2018 From: billk at iinet.net.au (Bill Kenworthy) Date: Sun, 28 Oct 2018 20:04:39 +0800 Subject: [Borgbackup] corrupted backup Message-ID: <2be831ef-278c-9c33-a80d-ba911e5c7fa0@iinet.net.au> I am getting the following error: Synchronizing chunks cache... Archives: 63, w/ cached Idx: 9, w/ outdated Idx: 0, w/o cached Idx: 54. Reading cached archive chunk index for lxc-20161230213958 ...958 Local Exception Traceback (most recent call last): ? File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4434, in main ??? exit_code = archiver.run(args) ? File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4366, in run ??? return set_ec(func(args)) ? File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 152, in wrapper ??? return method(self, args, repository=repository, **kwargs) ? File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 547, in do_create ??? cache_mode=args.files_cache_mode, ignore_inode=args.ignore_inode) as cache: ? File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 379, in __new__ ??? return local() ? File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 373, in local ??? lock_wait=lock_wait, cache_mode=cache_mode) ? File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 466, in __init__ ??? self.sync() ? File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 850, in sync ??? self.chunks = create_master_idx(self.chunks) ? File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 796, in create_master_idx ??? archive_chunk_idx = read_archive_index(archive_id, archive_name) ? File "/usr/lib64/python3.6/site-packages/borg/cache.py", line 739, in read_archive_index ??? archive_chunk_idx = ChunkIndex.read(fd, permit_compact=True) ? File "src/borg/hashindex.pyx", line 107, in borg.hashindex.IndexBase.read ? File "src/borg/hashindex.pyx", line 94, in borg.hashindex.IndexBase.__cinit__ ValueError: Could not read header (expected 18, but read 0 bytes) Platform: Linux lxc 4.14.65-gentoo #1 SMP Mon Aug 20 18:29:29 AWST 2018 x86_64 Linux: Gentoo Base System 2.4.1 Borg: 1.1.7? Python: CPython 3.6.5 PID: 22336? CWD: /root sys.argv: ['/usr/lib/python-exec/python3.6/borg', 'create', '-p', '-v', '--stats', '-C', 'lz4', '--exclude-caches', '--exclude-from', '/root/bbexcl', '--exclude', '/home/wdk/tmp', 'rattus.lan.localdomain:/mnt/backups/borg/lxc::lxc-{now:%Y%m%d%H%M%S}', '/'] SSH_ORIGINAL_COMMAND: None real??? 0m1.362s user??? 0m0.385s sys???? 0m0.056s I have run check --repair with no change. Is it recoverable?? I have a second backup of the system involved so I can lose this one and start again, but if it can be fixed I would like to do so. BillK From sebp at k-d-w.org Tue Oct 30 04:16:23 2018 From: sebp at k-d-w.org (=?UTF-8?Q?Sebastian_P=c3=b6lsterl?=) Date: Tue, 30 Oct 2018 09:16:23 +0100 Subject: [Borgbackup] Slow backup speed Message-ID: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> Hi, I'm using borg via borgmatic to backup of about 10TB of data. Since incremental backups take about 13 hours, I was wondering if there's any configuration option that can be used to speed things up, or if 13 hours is an expected duration. In total, the volume contains about 25 million files, but changes are rare. Backups are created daily and are carried out over a gigabit network connection with the client's disk mounted via NFS on the server running borg. Backups are created with --files-cache ctime,size and lz4 compression. Based on monitoring bandwidth usage, I would say that network bandwidth is not the bottleneck. Is there any configuration option I have been missing so far that could help to speed up daily backups? Thanks for your help! -- Sebastian From ndbecker2 at gmail.com Tue Oct 30 07:41:09 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 30 Oct 2018 07:41:09 -0400 Subject: [Borgbackup] How to tell if backup completed? Message-ID: Something crashed on my linux client during backup and I had to reboot. How can I tell if the backup completed? I'm assuming if it didn't I can just restart today's backup. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Tue Oct 30 07:48:07 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Tue, 30 Oct 2018 19:48:07 +0800 Subject: [Borgbackup] How to tell if backup completed? In-Reply-To: References: Message-ID: You can just restart. To know for sure, you can run borg list to see what the latest backup's time was or you can see the checkpoint files which would indicate an incomplete backup. On Tue, Oct 30, 2018 at 7:42 PM Neal Becker wrote: > Something crashed on my linux client during backup and I had to reboot. > How can I tell if the backup completed? I'm assuming if it didn't I can > just restart today's backup. > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabio.pedretti at unibs.it Tue Oct 30 13:36:05 2018 From: fabio.pedretti at unibs.it (Fabio Pedretti) Date: Tue, 30 Oct 2018 18:36:05 +0100 Subject: [Borgbackup] Slow backup speed In-Reply-To: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> References: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> Message-ID: You may want to use the following two options: --noatime --nobsdflags Also borg 1.2 should have some speed improvements when using millions of files, see for example this issue: https://github.com/borgbackup/borg/issues/3955 Il giorno mar 30 ott 2018 alle ore 09:55 Sebastian P?lsterl ha scritto: > Hi, > > I'm using borg via borgmatic to backup of about 10TB of data. Since > incremental backups take about 13 hours, I was wondering if there's any > configuration option that can be used to speed things up, or if 13 hours > is an expected duration. > > In total, the volume contains about 25 million files, but changes are > rare. Backups are created daily and are carried out over a gigabit > network connection with the client's disk mounted via NFS on the server > running borg. Backups are created with --files-cache ctime,size and lz4 > compression. Based on monitoring bandwidth usage, I would say that > network bandwidth is not the bottleneck. > > Is there any configuration option I have been missing so far that could > help to speed up daily backups? > > Thanks for your help! > > -- > Sebastian > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- ing. Fabio Pedretti Responsabile U.O.C. "Reti e Sistemi" http://www.unibs.it/organizzazione/amministrazione-centrale/servizio-servizi-ict/uoc-reti-e-sistemi Universit? degli Studi di Brescia Via Valotti, 9 - 25121 Brescia E-mail: fabio.pedretti at unibs.it -- Informativa sulla Privacy: http://www.unibs.it/node/8155 -------------- next part -------------- An HTML attachment was scrubbed... URL: From clickwir at gmail.com Tue Oct 30 13:59:07 2018 From: clickwir at gmail.com (Zack Coffey) Date: Tue, 30 Oct 2018 11:59:07 -0600 Subject: [Borgbackup] Slow backup speed In-Reply-To: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> References: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> Message-ID: I think you are right, it's not a bandwidth issue. But NFS dealing with that many small functions may be the bottleneck. I've seen NFS be the bottleneck many times when dealing with large amounts of small files or small changes. Big file transfers, it blazes. Tons of tiny ones, grinds to a halt. Some NFS mount tweaks can help that though. On Tue, Oct 30, 2018 at 2:55 AM Sebastian P?lsterl wrote: > Hi, > > I'm using borg via borgmatic to backup of about 10TB of data. Since > incremental backups take about 13 hours, I was wondering if there's any > configuration option that can be used to speed things up, or if 13 hours > is an expected duration. > > In total, the volume contains about 25 million files, but changes are > rare. Backups are created daily and are carried out over a gigabit > network connection with the client's disk mounted via NFS on the server > running borg. Backups are created with --files-cache ctime,size and lz4 > compression. Based on monitoring bandwidth usage, I would say that > network bandwidth is not the bottleneck. > > Is there any configuration option I have been missing so far that could > help to speed up daily backups? > > Thanks for your help! > > -- > Sebastian > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcpope at me.com Tue Oct 30 15:40:11 2018 From: marcpope at me.com (Marc Pope) Date: Tue, 30 Oct 2018 15:40:11 -0400 Subject: [Borgbackup] Slow backup speed In-Reply-To: References: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> Message-ID: What about running it from the client? NFS overhead is likely the problem. You could run it using a ssh command to the client from the server. > On Oct 30, 2018, at 1:59 PM, Zack Coffey wrote: > > I think you are right, it's not a bandwidth issue. But NFS dealing with that many small functions may be the bottleneck. > > I've seen NFS be the bottleneck many times when dealing with large amounts of small files or small changes. Big file transfers, it blazes. Tons of tiny ones, grinds to a halt. Some NFS mount tweaks can help that though. > >> On Tue, Oct 30, 2018 at 2:55 AM Sebastian P?lsterl wrote: >> Hi, >> >> I'm using borg via borgmatic to backup of about 10TB of data. Since >> incremental backups take about 13 hours, I was wondering if there's any >> configuration option that can be used to speed things up, or if 13 hours >> is an expected duration. >> >> In total, the volume contains about 25 million files, but changes are >> rare. Backups are created daily and are carried out over a gigabit >> network connection with the client's disk mounted via NFS on the server >> running borg. Backups are created with --files-cache ctime,size and lz4 >> compression. Based on monitoring bandwidth usage, I would say that >> network bandwidth is not the bottleneck. >> >> Is there any configuration option I have been missing so far that could >> help to speed up daily backups? >> >> Thanks for your help! >> >> -- >> Sebastian >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- An HTML attachment was scrubbed... URL: From sebp at k-d-w.org Tue Oct 30 17:01:22 2018 From: sebp at k-d-w.org (=?UTF-8?Q?Sebastian_P=c3=b6lsterl?=) Date: Tue, 30 Oct 2018 22:01:22 +0100 Subject: [Borgbackup] Slow backup speed In-Reply-To: References: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> Message-ID: <2e997c4e-02ba-de47-fd4a-3265e63a4375@k-d-w.org> Thanks for the suggestions so far. I'm going to experiment a little bit and see how things go. Unfortunately, I can't install software on the client, therefore I need to pull the backup. Were you referring to using sshfs instead of a NFS mount on the server? -- Sebastian Am 30.10.2018 um 20:40 schrieb Marc Pope: > What about running it from the client? NFS overhead is likely the > problem. You could run it using a ssh command to the client from the > server. > > On Oct 30, 2018, at 1:59 PM, Zack Coffey > wrote: > >> I think you are?right, it's not a bandwidth issue. But NFS dealing >> with that many small functions may be the bottleneck. >> >> I've seen NFS be the bottleneck many times when dealing with large >> amounts of small files or small changes. Big file transfers, it >> blazes. Tons of tiny ones, grinds to a halt. Some NFS mount tweaks can >> help that though. >> >> On Tue, Oct 30, 2018 at 2:55 AM Sebastian P?lsterl > > wrote: >> >> Hi, >> >> I'm using borg via borgmatic to backup of about 10TB of data. Since >> incremental backups take about 13 hours, I was wondering if >> there's any >> configuration option that can be used to speed things up, or if 13 >> hours >> is an expected duration. >> >> In total, the volume contains about 25 million files, but changes are >> rare. Backups are created daily and are carried out over a gigabit >> network connection with the client's disk mounted via NFS on the >> server >> running borg. Backups are created with --files-cache ctime,size >> and lz4 >> compression. Based on monitoring bandwidth usage, I would say that >> network bandwidth is not the bottleneck. >> >> Is there any configuration option I have been missing so far that >> could >> help to speed up daily backups? >> >> Thanks for your help! >> >> -- >> Sebastian >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup From maurice.libes at osupytheas.fr Thu Nov 1 08:43:25 2018 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 1 Nov 2018 13:43:25 +0100 Subject: [Borgbackup] advice on restoration Message-ID: <9d20b32b-7cc6-9c03-3e16-d63c69516a8c@osupytheas.fr> hello to all I have to restore a large amount of data (~ 1To) from a borg archive according to you, in term of duration, Is it better to do it with $ borg mount ... then cp or scp towards the NFS target disk volume or with $ borg extract or is it the same thing? II/ 2nd question I have already tried with borg extract but unfortunately, there have been a network problem and the restore process failed before the end So Is there a way to restart the restore without restoring the data that has already been recovered (a kind of restore rsync between the archive and the target disk) many thanks any advice is welcome for this restoration process ML -- M. LIBES Service Observation OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2971 bytes Desc: Signature cryptographique S/MIME URL: From ndbecker2 at gmail.com Thu Nov 1 14:06:48 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 1 Nov 2018 14:06:48 -0400 Subject: [Borgbackup] Zchunk File Format For Compressed Yet Efficient Deltas Nears 1.0 Release Message-ID: Perhaps this has some relevance for borg? https://www.phoronix.com/scan.php?page=news_item&px=Zchunk-1.0-Is-Near -------------- next part -------------- An HTML attachment was scrubbed... URL: From liori at exroot.org Thu Nov 1 16:57:59 2018 From: liori at exroot.org (Tomasz Melcer) Date: Thu, 1 Nov 2018 21:57:59 +0100 Subject: [Borgbackup] Slow backup speed In-Reply-To: References: <87645943-7a67-dc6a-a471-ea963437f779@k-d-w.org> Message-ID: <66a3cf2e-8ab3-1e0d-8174-7236b2f638f6@exroot.org> On 30.10.2018 18:59, Zack Coffey wrote: > I think you are?right, it's not a bandwidth issue. But NFS dealing with > that many small functions may be the bottleneck. This might be a latency issue, which could be hidden by running multiple I/O operations at the same time. Let say, scanning multiple directories in many threads. A quick test for this hypothesis would be comparing the time of two concurrent `find` commands to the time of a single `find` command doing the same. E.g.: echo 3 >/proc/sys/vm/drop_caches && time find dirA dirB -mount -uid 999 vs. echo 3 >/proc/sys/vm/drop_caches && time (find dirA -mount -uid 9999 & find dirB -mount -uid 9999 & wait) where `dirA` and `dirB` are two directories on that NFS. I tested this on a local HDD (the latter 4x slower, obviously) and on CIFS/samba (the latter ~20% faster). -- Tomasz Melcer From leone at diff.org Thu Nov 15 18:07:26 2018 From: leone at diff.org (Pietro Leone Pola Falletti di Villafalletto) Date: Fri, 16 Nov 2018 00:07:26 +0100 Subject: [Borgbackup] borg and vm backup Message-ID: <247e6de9-4e90-61b6-a598-83a8abd5758c@diff.org> Hallo, I'm using borg for backing up several things, a share, mail and vm. I copy data to be backed up on a dedicated nas, then I do the backup. After the backup I mount the last backup and check md5 hash of the files on the nas and files in the backup. I tried a restore of some file (the md5 check was ok) and I did md5sum of the file on the nas, the file in the backup and the restored file: b84496e57c6938a89ddfd6e0aa68fa4d /mnt/src/dati_rt/INVOICE KOSCHITZ1.jpg b84496e57c6938a89ddfd6e0aa68fa4d /mnt/borg/restore/src/20181115/mnt/src/dati_rt/INVOICE KOSCHITZ1.jpg b84496e57c6938a89ddfd6e0aa68fa4d /mnt/borg/restore/tgt/INVOICE KOSCHITZ1.jpg All are the same, so I should be sure the backup is OK. I tried the same for the vm vmdk: c61342bdebbcdbaeb37473d76e4a3e06 /mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk c61342bdebbcdbaeb37473d76e4a3e06 /mnt/borg/restore/src/20181115/mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk 79f8b416f4073e3e9a40da8989043073 /mnt/borg/restore/tgt/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk -rw-------. 1 root root 5368709120 Nov 15 02:40 /mnt/borg/restore/src/20181115/mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk -rw-------. 1 root root 5368709120 Nov 15 02:40 /mnt/borg/restore/tgt/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk -rw-------. 1 root root 5368709120 Nov 15 02:40 /mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk The last line is the restored vmdk, it differs from the others. I suppose because I use thin provisioning so the vmdk is a sparse file and maybe the restore process restore it as a thick provisioned file. I have imported the vm in esxi and it works, but how can I check a vm restore if the md5sum is different? Any idea? Thanks, Pietro. -- I will build myself a copper tower With four ways out and no way in But mine the glory, mine the power (So I chose AmigaOS, LinuxOS and OpenBSD) From tw at waldmann-edv.de Thu Nov 15 20:25:06 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 16 Nov 2018 02:25:06 +0100 Subject: [Borgbackup] borg and vm backup In-Reply-To: <247e6de9-4e90-61b6-a598-83a8abd5758c@diff.org> References: <247e6de9-4e90-61b6-a598-83a8abd5758c@diff.org> Message-ID: <0aff3bfe-0904-041f-3a7c-00d0ccaa4563@waldmann-edv.de> > c61342bdebbcdbaeb37473d76e4a3e06 > /mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk so i assume the above is the original file. > c61342bdebbcdbaeb37473d76e4a3e06 > /mnt/borg/restore/src/20181115/mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk this is the file accessed via "borg mount" ? > 79f8b416f4073e3e9a40da8989043073 > /mnt/borg/restore/tgt/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk and this the one extracted using "borg extract"? can you reproduce this? also on another machine? can you check the memory (memtest86+) and disk (smartctl -t long) of the machine that behaves strange? does sha512sum also give different results? > suppose because I use thin provisioning so the vmdk is a sparse file and > maybe the restore process restore it as a thick provisioned file. that is not relevant for md5sum, sparse ranges will still be read as zero bytes, same as if you have "on disk" zero bytes. if you do not find a hw or sw issue outside of borg, please file a bug on github and give more information there (see bug template). -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From leone at diff.org Fri Nov 16 08:03:38 2018 From: leone at diff.org (Pietro Leone Pola Falletti di Villafalletto) Date: Fri, 16 Nov 2018 14:03:38 +0100 Subject: [Borgbackup] borg and vm backup In-Reply-To: <0aff3bfe-0904-041f-3a7c-00d0ccaa4563@waldmann-edv.de> References: <247e6de9-4e90-61b6-a598-83a8abd5758c@diff.org> <0aff3bfe-0904-041f-3a7c-00d0ccaa4563@waldmann-edv.de> Message-ID: <8bdd7bf4-7ecc-1d59-1b5b-0229558c626c@diff.org> Sorry for the top quoting, but I wasted your time, I raised the flag too early. On 11/16/2018 2:25 AM, Thomas Waldmann wrote: >> c61342bdebbcdbaeb37473d76e4a3e06 >> /mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk > > so i assume the above is the original file. > Yes, correct. >> c61342bdebbcdbaeb37473d76e4a3e06 >> /mnt/borg/restore/src/20181115/mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk > > this is the file accessed via "borg mount" ? Yes. >> 79f8b416f4073e3e9a40da8989043073 >> /mnt/borg/restore/tgt/Soekris3/Soekris3-2018-11-15_01-15-01/Soekris3-1-flat.vmdk > > and this the one extracted using "borg extract"? Well, I simply copied (cp) the file from a mounted borg backup, otherwise I had to extract all the vm. > can you reproduce this? also on another machine? I tried to reproduce the issue and I could not, I tried extracting the file using several solutions (I tried more than one vm vmdk): 2763c447fd375a6da747f4906c4baf70 Soekris3-1-flat.vmdk-borg-extract 2763c447fd375a6da747f4906c4baf70 Soekris3-1-flat.vmdk-borg-extract-sparse 2763c447fd375a6da747f4906c4baf70 Soekris3-1-flat.vmdk-cp 2763c447fd375a6da747f4906c4baf70 Soekris3-1-flat.vmdk-rsync 2763c447fd375a6da747f4906c4baf70 Soekris3-1-flat.vmdk-rsync-sparse This is the md5 checksum of the original file: 2763c447fd375a6da747f4906c4baf70 /mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-16_01-15-01/Soekris3-1-flat.vmdk md5sum of the file in the mounted borg backup: 2763c447fd375a6da747f4906c4baf70 /mnt/borg/restore/src/mnt/src/vmbck/daily/Soekris3/Soekris3-2018-11-16_01-15-01/Soekris3-1-flat.vmdk All are the same so I made some error and I did not checked before asking. > can you check the memory (memtest86+) and disk (smartctl -t long) of the > machine that behaves strange? The borg server is a vm, I cannot stop the hypervisor for checking and I do not any issue on other vm. > > does sha512sum also give different results? >> suppose because I use thin provisioning so the vmdk is a sparse file and >> maybe the restore process restore it as a thick provisioned file. > > that is not relevant for md5sum, sparse ranges will still be read as > zero bytes, same as if you have "on disk" zero bytes. OK, thanks for the info. > if you do not find a hw or sw issue outside of borg, please file a bug > on github and give more information there (see bug template). > Thansk for your time, but seems that the issue is between the chair and the keyboard. Thanks, Pietro. -- I will build myself a copper tower With four ways out and no way in But mine the glory, mine the power (So I chose AmigaOS, LinuxOS and OpenBSD) From helge at monsternett.no Tue Nov 27 05:43:35 2018 From: helge at monsternett.no (Helge Milde) Date: Tue, 27 Nov 2018 11:43:35 +0100 Subject: [Borgbackup] Any way to make introspection of backups more performant? Message-ID: <20181127104335.GA29731@monsternett.no> I've been experimenting with moving from rdiff-backup to borgbackup. Mostly I'm in favor of switching (creating backups is a lot faster, it uses less disk space, allows for encryption, `borg mount` is an excellent tool, less chance of the backup becomming corrupt, allows for monthly/yearly backups, in active development etc..). The server I've been testing on is an e-mail server using ~2.7TB of disk space currently (so lots of small files). Extraction of data after a `borg mount` is very fast. The `borg mount` itself is very slow (8 minutes), and similarly `borg extract` is slower than the equivalent rdiff-backup command, but there seems to be a static time where borg builds an internal structure of the backup before the extraction actually happens. So rdiff-backup would be faster for extracting smaller amounts of data, but extracting a full backup seems like it will be a lot faster with borg, which is more important. The main problem I have is that gleaning any information from a backup (e.g. `borg list` or using `borg mount`) takes a lot of initial time. Whereas with rdiff-backup I can do `ls rdiff-backup/increments/var/www.*` to find the full history of a certain directory in the backup immediately (every creation, deletion or modification). This makes it possible to find exactly when the customer deleted their emails, or exactly when a site was hacked, and thus enable me to restore the most recent good backup of it pretty quickly. Is there anything that can enable this type of introspection of the full backup repository with borg? I guess this is just a design difference, in which case you can consider this a feature request :). But perhaps there's still something I can tweak (borg init/create commands, configuration, or give the backup server more than 8GB RAM) to make borg mount/list faster. I've realised I could save the output of the `borg create --filter=AME` commands as log files, and create tools that enables me to do similar tasks. On my backup repository, the `borg init` and the first `borg create` was done with borg 1.0.9-1~pbo8+1 (from jessie-backports). After that I changed to borg 1.1.7 from Github releases and did 3 more backups. I've read somewhere about borg having some bad defaults on earlier versions (something about chunk size?), though that might be talking about very old versions, so not sure if recreating the repository has any effect on performance. Here's some information about the repository & timing of some borg commands: http://dpaste.com/3DXGN4A From public at enkore.de Tue Nov 27 08:34:19 2018 From: public at enkore.de (Marian Beermann) Date: Tue, 27 Nov 2018 14:34:19 +0100 Subject: [Borgbackup] Any way to make introspection of backups more performant? In-Reply-To: <20181127104335.GA29731@monsternett.no> References: <20181127104335.GA29731@monsternett.no> Message-ID: Am 27.11.18 um 11:43 schrieb Helge Milde: > I've been experimenting with moving from rdiff-backup to borgbackup. > Mostly I'm in favor of switching (creating backups is a lot faster, it > uses less disk space, allows for encryption, `borg mount` is an > excellent tool, less chance of the backup becomming corrupt, allows for > monthly/yearly backups, in active development etc..). The server I've > been testing on is an e-mail server using ~2.7TB of disk space currently > (so lots of small files). > > Extraction of data after a `borg mount` is very fast. The `borg mount` > itself is very slow (8 minutes), and similarly `borg extract` is slower > than the equivalent rdiff-backup command, but there seems to be a static > time where borg builds an internal structure of the backup before the > extraction actually happens. So rdiff-backup would be faster for > extracting smaller amounts of data, but extracting a full backup seems > like it will be a lot faster with borg, which is more important When you extract part of an archive (e.g. borg extract foo::bar mail/user/1), then Borg performs a linear scan of all files in the archive and matches each against the specified arguments. So if you extract a single directory there tends to be a prelude, which is Borg scanning all the files appearing before the directory in the archive, then the files are extracted and then a delay before it exits, which is Borg scanning all the files appearing after the directory in the archive. > The main problem I have is that gleaning any information from a backup > (e.g. `borg list` or using `borg mount`) takes a lot of initial time. > Whereas with rdiff-backup I can do `ls > rdiff-backup/increments/var/www.*` to find the full history of a certain > directory in the backup immediately (every creation, deletion or > modification). This makes it possible to find exactly when the customer > deleted their emails, or exactly when a site was hacked, and thus enable > me to restore the most recent good backup of it pretty quickly. Is there > anything that can enable this type of introspection of the full backup > repository with borg? No, there are no indices to make lookups like that. Both Borg and any additional tool (that doesn't explicitly manage such a change index) needs to linearly scan the archives involved. "Scanning the archive" means reading all the serialized (special Borg format) metadata about all files in the archive. For small files this may be a significant fraction of the total data size. > I guess this is just a design difference, in which case you can consider > this a feature request :). But perhaps there's still something I can > tweak (borg init/create commands, configuration, or give the backup > server more than 8GB RAM) to make borg mount/list faster. I've realised > I could save the output of the `borg create --filter=AME` commands as > log files, and create tools that enables me to do similar tasks. Yes. > On my backup repository, the `borg init` and the first `borg create` was > done with borg 1.0.9-1~pbo8+1 (from jessie-backports). After that I > changed to borg 1.1.7 from Github releases and did 3 more backups. I've > read somewhere about borg having some bad defaults on earlier versions > (something about chunk size?), though that might be talking about very > old versions, so not sure if recreating the repository has any effect on > performance. That's about going from 0.x or Attic to 1.x. Cheers, Marian From tw at waldmann-edv.de Tue Nov 27 09:12:34 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 27 Nov 2018 15:12:34 +0100 Subject: [Borgbackup] Any way to make introspection of backups more performant? In-Reply-To: References: <20181127104335.GA29731@monsternett.no> Message-ID: <7ce3745a-fcd6-52e9-7762-5a4eeef6cbab@waldmann-edv.de> Thanks Marian for explaining! > When you extract part of an archive (e.g. borg extract foo::bar> mail/user/1), then Borg performs a linear scan of all files in the> archive and matches each against the specified arguments. So if you> extract a single directory there tends to be a prelude, which is Borg> scanning all the files appearing before the directory in the archive,> then the files are extracted and then a delay before it exits, which is> Borg scanning all the files appearing after the directory in the archive. That "scanning after" appears a bit strange, but is due to the fact that there is no "directory object" with a list of contained items (files, dirs, ...) in the archive. So directories can only be implicitly derived from the full pathnames stored with each item. We maybe could optimize a bit though based on our knowledge of directory traversal order by detecting (based on the pathnames seen) when we "leave" a directory. Not sure if we already do that. OTOH, maybe we rather don't as this "known traversal order" might get mixed up again once we introduce multithreading / parallel processing of files. From tw at waldmann-edv.de Tue Nov 27 09:24:11 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 27 Nov 2018 15:24:11 +0100 Subject: [Borgbackup] Any way to make introspection of backups more performant? In-Reply-To: <7ce3745a-fcd6-52e9-7762-5a4eeef6cbab@waldmann-edv.de> References: <20181127104335.GA29731@monsternett.no> <7ce3745a-fcd6-52e9-7762-5a4eeef6cbab@waldmann-edv.de> Message-ID: <7df78ea7-ff9e-e5fe-9915-b0159ecdd46a@waldmann-edv.de> Embarassing how badly formatted my post looked on the list, while it looked good in Thunderbird. Sorry about that. From dchebota at gmu.edu Mon Dec 3 20:20:06 2018 From: dchebota at gmu.edu (Dmitri Chebotarov) Date: Tue, 4 Dec 2018 01:20:06 +0000 Subject: [Borgbackup] SSH performance on Borg server Message-ID: Hello Has anyone tried to use HPN patched SSH on Borg server? If you did, can you share your experience and if it helped to speed up backups? With multiple simultaneous backups the time needed to complete each backup increases significantly. I?m trying to find the bottleneck, and the server?s resources are not used much (cpu/mem/net), at least nothing stands out. Thank you, -- Dmitri Chebotarov George Mason University, 4400 University Drive, Fairfax, VA, 22030 Phone: (703) 993-6175 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ngoonee.talk at gmail.com Wed Dec 5 20:39:31 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Thu, 6 Dec 2018 09:39:31 +0800 Subject: [Borgbackup] borg-import - use to merge backups? Message-ID: I've been using borg for just over a month now to test it out, and I think I'm going to delete my old rsnapshot archives (which go back 2 years). Ideally I'd like to convert them to borg archives instead of just deleting them, so I just found borg-import[1] which looks like it'll do what I need. However it seems to be geared towards working on an empty repo. How would it fare if used on my existing borg repo (which I've been using for the past month)? In particular, how would the dating of the archives work (is there a way to 'set' the date of an archive to not be the date the archive was generated?)? Also, I'm going to do some surgery on the rsnapshot archives first. I've always hated how the paths were from root, so the snapshot directory would have something like month.3/mnt/HDD/path/to/actual/folder/through/the/entire/tree, and there's some directories I no longer care about that I'll just delete. Would such modifications make a big difference to borg-import? Should I, instead of using borg-import, be looking at manually creating archives? [1] - https://github.com/borgbackup/borg-import -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at enkore.de Thu Dec 6 13:01:34 2018 From: public at enkore.de (Marian Beermann) Date: Thu, 6 Dec 2018 19:01:34 +0100 Subject: [Borgbackup] borg-import - use to merge backups? In-Reply-To: References: Message-ID: <894ec611-61fc-7bbf-8598-730da16069e1@enkore.de> Am 06.12.18 um 02:39 schrieb Oon-Ee Ng: > I've been using borg for just over a month now to test it out, and I > think I'm going to delete my old rsnapshot archives (which go back 2 years). > > Ideally I'd like to convert them to borg archives instead of just > deleting them, so I just found borg-import[1] which looks like it'll do > what I need. > > However it seems to be geared towards working on an empty repo. How > would it fare if used on my existing borg repo (which I've been using > for the past month)? In particular, how would the dating of the archives > work (is there a way to 'set' the date of an archive to not be the date > the archive was generated?)? borg-import creates archives using "borg create". That doesn't impact existing archives. The timestamp of archives is the mtime of the snapshot directory. > Also, I'm going to do some surgery on the rsnapshot archives first. I've > always hated how the paths were from root, so the snapshot directory > would have something like > month.3/mnt/HDD/path/to/actual/folder/through/the/entire/tree, and > there's some directories I no longer care about that I'll just delete. > Would such modifications make a big difference to borg-import? I think it should be fairly straightforward to patch borg-import to have it do that; it doesn't have an option out of the box. You can pass excludes to the "borg create" command line, though, so if all you want is to exclude some parts of the existing snapshots, then that's no problem. > Should I, instead of using borg-import, be looking at manually creating > archives? borg-import basically automates invoking "borg create" for each snapshot. So either will pretty much have the same outcome. Cheers, Marian From tw at waldmann-edv.de Sun Dec 9 07:53:28 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 9 Dec 2018 13:53:28 +0100 Subject: [Borgbackup] borgbackup 1.1.8 released! Message-ID: see there: https://github.com/borgbackup/borg/releases/tag/1.1.8 From shacky83 at gmail.com Mon Dec 10 03:29:54 2018 From: shacky83 at gmail.com (shacky) Date: Mon, 10 Dec 2018 09:29:54 +0100 Subject: [Borgbackup] Backup big volumes to multiple USB drives Message-ID: Hi. I have some big volumes (>10 Tb) which I need to backup on multiple external USB drives (8 Tb each). These volumes are incremental or something like "round robin": no files are deleted but files are only added to them. When we have no space on them, old data is moved to some offline volumes but they have not to be considered as "deleted". I am wondering if Borg Backup could help me to backup these files. It should work with a single connected USB drive but remembering what was backupped to other USB drives (which cannot be connected in the same time). And if I look for a file it should tell me to what USB drive it was backupped so I can connect it and take it out. Could you help me? If Borg Backup is not a solution for this, could you advise me another solution, please? Thank you very much! Bye -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Mon Dec 10 03:43:32 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 10 Dec 2018 09:43:32 +0100 Subject: [Borgbackup] Backup big volumes to multiple USB drives In-Reply-To: References: Message-ID: <592107b3-1208-9a82-576d-5496d22a8963@waldmann-edv.de> There is no support in borg for repositories that are on multiple filesystems. > I have some big volumes (>10 Tb) which I need to backup on multiple > external USB drives (8 Tb each). Depending on how much data it really is and how compressible it is, it might fit on one drive. > If Borg Backup is not a solution for this, could you advise me another > solution, please? I guess there are some tools, but I do not have any experience with them. From public at enkore.de Mon Dec 10 07:19:44 2018 From: public at enkore.de (Marian Beermann) Date: Mon, 10 Dec 2018 13:19:44 +0100 Subject: [Borgbackup] Backup big volumes to multiple USB drives In-Reply-To: References: Message-ID: tar(1) and split(1) might be what you're looking for. Note that you can concatenate tar files to combine them (see tar --ignore-zeros). Cheers, Marian Am 10.12.18 um 09:29 schrieb shacky: > Hi. > I have some big volumes (>10 Tb) which I need to backup on multiple > external USB drives (8 Tb each). > > These volumes are incremental or something like "round robin": no files > are deleted but files are only added to them. When we have no space on > them, old data is moved to some offline volumes but they have not to be > considered as "deleted". > > I am wondering if Borg Backup could help me to backup these files. It > should work with a single connected USB drive but remembering what was > backupped to other USB drives (which cannot be connected in the same > time). And if I look for a file it should tell me to what USB drive it > was backupped so I can connect it and take it out. > > Could you help me? > If Borg Backup is not a solution for this, could you advise me another > solution, please? > > Thank you very much! > Bye > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From plattrap at lholding.net Mon Dec 10 15:51:11 2018 From: plattrap at lholding.net (Lawrence Holding) Date: Tue, 11 Dec 2018 09:51:11 +1300 Subject: [Borgbackup] Backup big volumes to multiple USB drives In-Reply-To: References: Message-ID: Try backing up all the big volumes to a single usb drive. The compression and deduplication may allow them all to fit. But if each big volume has unique data, then this would not work. Or pair a big volume with a usb drive, and always backup to each other? Enjoy, > On 10/12/2018, at 21:29, shacky wrote: > > Hi. > I have some big volumes (>10 Tb) which I need to backup on multiple external USB drives (8 Tb each). > > These volumes are incremental or something like "round robin": no files are deleted but files are only added to them. When we have no space on them, old data is moved to some offline volumes but they have not to be considered as "deleted". > > I am wondering if Borg Backup could help me to backup these files. It should work with a single connected USB drive but remembering what was backupped to other USB drives (which cannot be connected in the same time). And if I look for a file it should tell me to what USB drive it was backupped so I can connect it and take it out. > > Could you help me? > If Borg Backup is not a solution for this, could you advise me another solution, please? > > Thank you very much! > Bye > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From dchebota at gmu.edu Tue Dec 11 11:34:52 2018 From: dchebota at gmu.edu (Dmitri Chebotarov) Date: Tue, 11 Dec 2018 16:34:52 +0000 Subject: [Borgbackup] [Errno 2] No such file or directory: Message-ID: Hello Is is possible to ignore missing files during backup? F.e. I get these error messages: /graylog/elasticsearch/data/nodes/0/indices/1drnTgR7Rgq6MtI2zv4WRQ/0/index/_dkrn.cfe: stat: [Errno 2] No such file or directory: '/graylog/elasticsearch/data/nodes/0/indices/1drnTgR7Rgq6MtI2zv4WRQ/0/index/_dkrn.cfe' /graylog/elasticsearch/data/nodes/0/indices/1drnTgR7Rgq6MtI2zv4WRQ/0/index/_dkrn.si: stat: [Errno 2] No such file or directory: 'borg create' exists with non-0 code ($? != 0). I couldn't find any options for 'borg create' to ignore these errors. As a workaround I'm excluding /graylog/elasticsearch/data/nodes/0/indices/* dir from backup, but I've seen it's happening on other dirs as well... Thank you, -- Dmitri Chebotarov. George Mason University, 4400 University Drive, Fairfax, VA, 22030 GPG Public key# 5E19F14D: [https://goo.gl/SlE8tj] From devzero at web.de Tue Dec 11 13:53:30 2018 From: devzero at web.de (devzero at web.de) Date: Tue, 11 Dec 2018 19:53:30 +0100 Subject: [Borgbackup] [Errno 2] No such file or directory: In-Reply-To: References: Message-ID: yes please - on rsync this is a warning, not an error. so maybe borg could handle that in the same way... file has vanished: "/var/opt/mssql/log/log_63001.trc" rsync warning: some files vanished before they could be transferred (code 24) at main.c(1650) [generator=3.1.2] > Gesendet: Dienstag, 11. Dezember 2018 um 17:34 Uhr > Von: "Dmitri Chebotarov" > An: "borgbackup at python.org" > Betreff: [Borgbackup] [Errno 2] No such file or directory: > > Hello > > Is is possible to ignore missing files during backup? > F.e. I get these error messages: > > /graylog/elasticsearch/data/nodes/0/indices/1drnTgR7Rgq6MtI2zv4WRQ/0/index/_dkrn.cfe: stat: [Errno 2] No such file or directory: '/graylog/elasticsearch/data/nodes/0/indices/1drnTgR7Rgq6MtI2zv4WRQ/0/index/_dkrn.cfe' > /graylog/elasticsearch/data/nodes/0/indices/1drnTgR7Rgq6MtI2zv4WRQ/0/index/_dkrn.si: stat: [Errno 2] No such file or directory: > > 'borg create' exists with non-0 code ($? != 0). > > I couldn't find any options for 'borg create' to ignore these errors. > As a workaround I'm excluding /graylog/elasticsearch/data/nodes/0/indices/* dir from backup, but I've seen it's happening on other dirs as well... > > Thank you, > -- > Dmitri Chebotarov. > George Mason University, > 4400 University Drive, > Fairfax, VA, 22030 > GPG Public key# 5E19F14D: [https://goo.gl/SlE8tj] > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From eric at in3x.io Tue Dec 11 16:02:38 2018 From: eric at in3x.io (Eric S. Johansson) Date: Tue, 11 Dec 2018 16:02:38 -0500 Subject: [Borgbackup] Backup big volumes to multiple USB drives In-Reply-To: References: Message-ID: <441d1775-116b-8495-f544-1395c2502352@in3x.io> On 12/10/2018 3:29 AM, shacky wrote: > Hi. > I have some big volumes (>10 Tb) which I need to backup on multiple > external USB drives (8 Tb each). > > These volumes are incremental or something like "round robin": no > files are deleted but files are only added to them. When we have no > space on them, old data is moved to some offline volumes but they have > not to be considered as "deleted". > > I am wondering if Borg Backup could help me to backup these files. It > should work with a single connected USB drive but remembering what was > backupped to other USB drives (which cannot be connected in the same > time). And if I look for a file it should tell me to what USB drive it > was backupped so I can connect it and take it out. How offended would you be if I said "oh dear God, no. Just no"? I have this reaction is because (scar tissue) external USB drives are not very reliable. also as others have indicated, this is not something Borg can do.? do you have a reason why a local NAS with enough storage would not be acceptable as a solution? are you trying to take the backup data off-site using these external drives? Are you open to suggestions from the mailing list on better ways to handle your backup strategy? -- Eric S. Johansson eric at in3x.io http://www.in3x.io 978-512-0272 From liori at exroot.org Tue Dec 11 18:30:30 2018 From: liori at exroot.org (Tomasz Melcer) Date: Wed, 12 Dec 2018 00:30:30 +0100 Subject: [Borgbackup] Backup big volumes to multiple USB drives In-Reply-To: References: Message-ID: On 10.12.2018 09:29, shacky wrote: > Could you help me? > If Borg Backup is not a solution for this, could you advise me another > solution, please? I don't see borg being useful in this case. However, `git annex` [1] looks like it might be a nice base to build a kind of redundant storage for your data, especially if you can make your main volumes also be `git annex` repositories. It has its limitations, though. [1] https://git-annex.branchable.com/ -- Tomasz Melcer From catscrash at catscrash.de Wed Dec 12 02:28:22 2018 From: catscrash at catscrash.de (Catscrash) Date: Wed, 12 Dec 2018 08:28:22 +0100 Subject: [Borgbackup] Backup MySQL in Borg Message-ID: Hi, I'm looking for some best practices to backup mysql databases. I see multiple scenarios and issues with each of them, so far I didn't find an ideal way and am looking for your input 1. Backing up just the datadir files * Advantages: o Full use of borg deduplication o No additional space needed on source system * Disadvantages: o Restoring gets a lot harder, the data can't be used across different versions of mysql/mariadb, risk of issues when restoring are a lot harder o can't restore single databases / tables 2. Backing up uncompressed mysqldump file * Advantages: o Use of borg deduplication, especially for bigger dumps, if only some tables change, deduplication should take care of the rest o easily restorable * Disadvantages o There must be enough space to store the complete dump uncompressed, this can be an issue with databases in the region of 600gb+, it's not possible to keep that much space on all the systems 3. Backing up compressed (e.g. gz) mysqldump file * Advantages: o Much less space needed on source system, as mysqldumps are highly compressable o easily restorable * Disadvantages o Deduplication not possible anymore, which makes the backup unusable big after a short time How are you solving this? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From liori at exroot.org Wed Dec 12 02:42:51 2018 From: liori at exroot.org (Tomasz Melcer) Date: Wed, 12 Dec 2018 08:42:51 +0100 Subject: [Borgbackup] Backup MySQL in Borg In-Reply-To: References: Message-ID: <4fa7978e-beff-2fd6-a612-aaf10cda5cc5@exroot.org> On 12.12.2018 08:28, Catscrash wrote: > o There must be enough space to store the complete dump > uncompressed, this can be an issue with databases in the > region of 600gb+, it's not possible to keep that much space > on all the systems I'm not using borg to back up databases, so I can't recommend any of these options, but at least this point is manageable: you can pipe the output of `mysqldump` into borg without storing the temporary file anywhere by giving `-` as the path to store. mysqldump ? | borg create /your/repository::mysqldump-backup - -- Tomasz Melcer From jasper at knockaert.nl Wed Dec 12 02:41:09 2018 From: jasper at knockaert.nl (Jasper Knockaert) Date: Wed, 12 Dec 2018 08:41:09 +0100 Subject: [Borgbackup] Backup MySQL in Borg In-Reply-To: References: Message-ID: Borg can use stdin for input. There is no need to store a full uncompressed dump, just redirect it to Borg. Jasper Catscrash schreef op 12 december 2018 08:28:22 CET: >Hi, > >I'm looking for some best practices to backup mysql databases. > >I see multiple scenarios and issues with each of them, so far I didn't >find an ideal way and am looking for your input > > 1. Backing up just the datadir files > * Advantages: > o Full use of borg deduplication > o No additional space needed on source system > * Disadvantages: > o Restoring gets a lot harder, the data can't be used across > different versions of mysql/mariadb, risk of issues when > restoring are a lot harder > o can't restore single databases / tables > 2. Backing up uncompressed mysqldump file > * Advantages: > o Use of borg deduplication, especially for bigger dumps, if > only some tables change, deduplication should take care of > the rest > o easily restorable > * Disadvantages > o There must be enough space to store the complete dump > uncompressed, this can be an issue with databases in the > region of 600gb+, it's not possible to keep that much space > on all the systems > 3. Backing up compressed (e.g. gz) mysqldump file > * Advantages: > o Much less space needed on source system, as mysqldumps are > highly compressable > o easily restorable > * Disadvantages > o Deduplication not possible anymore, which makes the backup > unusable big after a short time > > >How are you solving this? > > >Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From catscrash at catscrash.de Wed Dec 12 02:55:58 2018 From: catscrash at catscrash.de (Catscrash) Date: Wed, 12 Dec 2018 08:55:58 +0100 Subject: [Borgbackup] Backup MySQL in Borg In-Reply-To: <4fa7978e-beff-2fd6-a612-aaf10cda5cc5@exroot.org> References: <4fa7978e-beff-2fd6-a612-aaf10cda5cc5@exroot.org> Message-ID: Thanks Tomasz and Jasper for the stdin info, that's good to know. Is it also possible to use the pipe as part of another repository, so that the pipe input appears as one file in a specific path and as part of the rest of the backed up data, or would I definitely need to have a separate repository for this? Thanks a lot! Am 12.12.18 um 08:42 schrieb Tomasz Melcer: > On 12.12.2018 08:28, Catscrash wrote: >> ????????? o There must be enough space to store the complete dump >> ??????????? uncompressed, this can be an issue with databases in the >> ??????????? region of 600gb+, it's not possible to keep that much space >> ??????????? on all the systems > > I'm not using borg to back up databases, so I can't recommend any of > these options, but at least this point is manageable: you can pipe the > output of `mysqldump` into borg without storing the temporary file > anywhere by giving `-` as the path to store. > > mysqldump ? | borg create /your/repository::mysqldump-backup - > > From dirk at deimeke.net Wed Dec 12 03:17:29 2018 From: dirk at deimeke.net (Dirk Deimeke) Date: Wed, 12 Dec 2018 08:17:29 +0000 Subject: [Borgbackup] Backup MySQL in Borg In-Reply-To: References: Message-ID: December 12, 2018 8:28 AM, "Catscrash" wrote: Hi! > How are you solving this? Depending on the restore time requirements I have two different approaches. 1. Using MyDumper - https://github.com/maxbube/mydumper MyDumper does a parallelized dump of data, running much faster than the original mysqldump. The tool myloader does restore parallelized as well. 2. Using LVM snapshots - stop the database for a minimal amount of time (milliseconds) - create a snapshot of the data volume carrying /var/lib/mysql - start the database again - mount the snapshot readonly - backup the snapshot - unmount the snapshot - delete the snapshot This requires free space in the volume group. Cheers Dirk -- https://d5e.org From eric at in3x.io Wed Dec 12 10:22:58 2018 From: eric at in3x.io (Eric S. Johansson) Date: Wed, 12 Dec 2018 10:22:58 -0500 Subject: [Borgbackup] best way to back up running KVM images Message-ID: <8edaa60f-9b8a-0b5c-cb7a-457f4ced38d2@in3x.io> I have a machine running 3 KVM guests.? I can't stop or suspend them for a backup (one is a domain controller). I am on zfs so? I may be able to do a suspend operation / snapshot trick.? I've also seen a clone, backup the original , then merge back in the clone/ trick as well but I must be missing something because I never got that to work.? So, any suggestions? -- Eric S. Johansson eric at in3x.io http://www.in3x.io 978-512-0272 From public at enkore.de Wed Dec 12 10:54:37 2018 From: public at enkore.de (Marian Beermann) Date: Wed, 12 Dec 2018 16:54:37 +0100 Subject: [Borgbackup] best way to back up running KVM images In-Reply-To: <8edaa60f-9b8a-0b5c-cb7a-457f4ced38d2@in3x.io> References: <8edaa60f-9b8a-0b5c-cb7a-457f4ced38d2@in3x.io> Message-ID: <87100d64-aa4b-9757-8fc5-b58184f18f8d@enkore.de> With virtual machines you want at least crash-consistency for your backups, i.e. you should snapshot the disk images. qcow2 supports this. Snapshotting the FS the disk images are on may / may not be equivalent. Probably not equivalent. If you are running databases inside VMs I'd recommend to additionally use the DB-specific dump tools. Cheers, Marian Am 12.12.18 um 16:22 schrieb Eric S. Johansson: > I have a machine running 3 KVM guests.? I can't stop or suspend them for > a backup (one is a domain controller). I am on zfs so? I may be able to > do a suspend operation / snapshot trick.? I've also seen a clone, backup > the original , then merge back in the clone/ trick as well but I must be > missing something because I never got that to work.? > > So, any suggestions? > From mateusz.kijowski at gmail.com Wed Dec 12 12:29:16 2018 From: mateusz.kijowski at gmail.com (Mateusz Kijowski) Date: Wed, 12 Dec 2018 18:29:16 +0100 Subject: [Borgbackup] Backup MySQL in Borg In-Reply-To: References: Message-ID: Although I'm not using borg to backup MySQL databases you could also consider xtrabackup [1] for doing a binary backup of the database without downtime. Cheers, Mateusz [1] https://www.percona.com/doc/percona-xtrabackup/8.0/index.html or https://www.percona.com/doc/percona-xtrabackup/2.4/index.html ?r., 12 gru 2018 o 09:26 Dirk Deimeke napisa?(a): > > December 12, 2018 8:28 AM, "Catscrash" wrote: > > Hi! > > > How are you solving this? > > Depending on the restore time requirements I have two different approaches. > > 1. Using MyDumper - https://github.com/maxbube/mydumper > > MyDumper does a parallelized dump of data, running much faster than the original mysqldump. The tool myloader does restore parallelized as well. > > 2. Using LVM snapshots > > - stop the database for a minimal amount of time (milliseconds) > - create a snapshot of the data volume carrying /var/lib/mysql > - start the database again > - mount the snapshot readonly > - backup the snapshot > - unmount the snapshot > - delete the snapshot > > This requires free space in the volume group. > > Cheers > > Dirk > > -- > https://d5e.org > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From devzero at web.de Fri Dec 14 13:46:35 2018 From: devzero at web.de (devzero at web.de) Date: Fri, 14 Dec 2018 19:46:35 +0100 Subject: [Borgbackup] zstd in production ? Message-ID: hello, i tink of switching compression to zstd and re-creating all borg archives for space-saving. is anybody unsing zstd in production including recovery-testing ? as it's not standard compression, i'm unsure if it's worh the risk just for saving about 1tb of diskspace... regards roland From devzero at web.de Tue Dec 18 20:52:25 2018 From: devzero at web.de (devzero at web.de) Date: Wed, 19 Dec 2018 02:52:25 +0100 Subject: [Borgbackup] faq entry regarding "added" status for an unchanged file Message-ID: i think "files that are backed up from a snapshot" could need some better explanation. what is meant here? what "snapshots"? are you speaking of virtual machine snashots? what about some additional hint in the log "hey, i'm doing something special here...." ? files which "change during save" are always inconsistent - rsync for example warns about this... i don't get the real point what's being adressed with this feature and why borg does handle things different from rsync. why is it default behaviour to always backup the file with the latest mtime - and why not adding a special backup option for a special use-case? i'm asking this because i accidentally opened a bugticket because i "observed something strange i could not explain"... regards roland I am seeing ?A? (added) status for an unchanged file!? The files cache is used to determine whether Borg already ?knows? / has backed up a file and if so, to skip the file from chunking. It does intentionally not contain files that have a modification time (mtime) same as the newest mtime in the created archive. So, if you see an ?A? status for unchanged file(s), they are likely the files with the most recent mtime in that archive. This is expected: it is to avoid data loss with files that are backed up from a snapshot and that are immediately changed after the snapshot (but within mtime granularity time, so the mtime would not change). Without the code that removes these files from the files cache, the change that happened right after the snapshot would not be contained in the next backup as Borg would think the file is unchanged. This does not affect deduplication, the file will be chunked, but as the chunks will often be the same and already stored in the repo (except in the above mentioned rare condition), it will just re-use them as usual and not store new data chunks. If you want to avoid unnecessary chunking, just create or touch a small or empty file in your backup source file set (so that one has the latest mtime, not your 50GB VM disk image) and, if you do snapshots, do the snapshot after that. Since only the files cache is used in the display of files status, those files are reported as being added when, really, chunks are already used. From ngoonee.talk at gmail.com Tue Dec 18 22:36:30 2018 From: ngoonee.talk at gmail.com (Oon-Ee Ng) Date: Wed, 19 Dec 2018 11:36:30 +0800 Subject: [Borgbackup] faq entry regarding "added" status for an unchanged file In-Reply-To: References: Message-ID: The explanation is already clear, it just says that one file (the latest file) will have its' mtime used to determine the archive's mtime. This means that that file (and any others with the same or greater mtime) will be chunked in the next run. On Wed, Dec 19, 2018 at 9:52 AM wrote: > i think "files that are backed up from a snapshot" could need some better > explanation. > what is meant here? what "snapshots"? > are you speaking of virtual machine snashots? > > what about some additional hint in the log "hey, i'm doing something > special here...." ? > > files which "change during save" are always inconsistent - rsync for > example warns about this... > > i don't get the real point what's being adressed with this feature and why > borg does handle things different from rsync. > > why is it default behaviour to always backup the file with the latest > mtime - and why not adding a special backup option for a special use-case? > > i'm asking this because i accidentally opened a bugticket because i > "observed something strange i could not explain"... > > regards > roland > > > I am seeing ?A? (added) status for an unchanged file!? > > The files cache is used to determine whether Borg already ?knows? / has > backed up a file and if so, to skip the file from chunking. It does > intentionally not contain files that have a modification time (mtime) same > as the newest mtime in the created archive. > > So, if you see an ?A? status for unchanged file(s), they are likely the > files with the most recent mtime in that archive. > > This is expected: it is to avoid data loss with files that are backed up > from a snapshot and that are immediately changed after the snapshot (but > within mtime granularity time, so the mtime would not change). Without the > code that removes these files from the files cache, the change that > happened right after the snapshot would not be contained in the next backup > as Borg would think the file is unchanged. > > This does not affect deduplication, the file will be chunked, but as the > chunks will often be the same and already stored in the repo (except in the > above mentioned rare condition), it will just re-use them as usual and not > store new data chunks. > > If you want to avoid unnecessary chunking, just create or touch a small or > empty file in your backup source file set (so that one has the latest > mtime, not your 50GB VM disk image) and, if you do snapshots, do the > snapshot after that. > > Since only the files cache is used in the display of files status, those > files are reported as being added when, really, chunks are already used. > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From gait at atcomputing.nl Wed Dec 19 02:48:29 2018 From: gait at atcomputing.nl (Gerrit A. Smit) Date: Wed, 19 Dec 2018 08:48:29 +0100 Subject: [Borgbackup] faq entry regarding "added" status for an unchanged file In-Reply-To: References: Message-ID: devzero at web.de schreef op 19-12-18 om 02:52: > i think "files that are backed up from a snapshot" could need some better explanation. > what is meant here? what "snapshots"? > are you speaking of virtual machine snashots? Borg Backup is about files, not about VMs. So a snapshot is about file(system)snapshots. Greetings,, Gerrit -------------- next part -------------- An HTML attachment was scrubbed... URL: From devzero at web.de Wed Dec 19 03:51:40 2018 From: devzero at web.de (devzero at web.de) Date: Wed, 19 Dec 2018 09:51:40 +0100 Subject: [Borgbackup] faq entry regarding "added" status for an unchanged file In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From devzero at web.de Wed Dec 19 04:10:48 2018 From: devzero at web.de (devzero at web.de) Date: Wed, 19 Dec 2018 10:10:48 +0100 Subject: [Borgbackup] faq entry regarding "added" status for an unchanged file In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From devzero at web.de Wed Dec 19 14:50:05 2018 From: devzero at web.de (devzero at web.de) Date: Wed, 19 Dec 2018 20:50:05 +0100 Subject: [Borgbackup] Fw: faq entry regarding "added" status for an unchanged file References: Message-ID: spent time on thinking on this, spoke with my collueage - we both don't get the point. we both care about file/data integrity, so we find this question interesting. could somebody explain in other words given an example? i do snapshot "snap1" of filesystem. lets say next snapshot (snap2) is done seconds later on. then i backup files from snap1. then do backup from snap2. files which changed in between snap1 and snap2 have different mtime in snap2. ok. these are the files being chunked/re-added, all others getting skipped because of borg-cache entries... so why is only _one_ single file being handled in a special way? (as multiple files could have changed...) sorry, maybe i'm too dumb... regards roland > Gesendet: Mittwoch, 19. Dezember 2018 um 02:52 Uhr > Von: devzero at web.de > An: "borgbackuppython.org" > Betreff: faq entry regarding "added" status for an unchanged file > > i think "files that are backed up from a snapshot" could need some better explanation. > what is meant here? what "snapshots"? > are you speaking of virtual machine snashots? > > what about some additional hint in the log "hey, i'm doing something special here...." ? > > files which "change during save" are always inconsistent - rsync for example warns about this... > > i don't get the real point what's being adressed with this feature and why borg does handle things different from rsync. > > why is it default behaviour to always backup the file with the latest mtime - and why not adding a special backup option for a special use-case? > > i'm asking this because i accidentally opened a bugticket because i "observed something strange i could not explain"... > > regards > roland > > > I am seeing ?A? (added) status for an unchanged file!? > > The files cache is used to determine whether Borg already ?knows? / has backed up a file and if so, to skip the file from chunking. It does intentionally not contain files that have a modification time (mtime) same as the newest mtime in the created archive. > > So, if you see an ?A? status for unchanged file(s), they are likely the files with the most recent mtime in that archive. > > This is expected: it is to avoid data loss with files that are backed up from a snapshot and that are immediately changed after the snapshot (but within mtime granularity time, so the mtime would not change). Without the code that removes these files from the files cache, the change that happened right after the snapshot would not be contained in the next backup as Borg would think the file is unchanged. > > This does not affect deduplication, the file will be chunked, but as the chunks will often be the same and already stored in the repo (except in the above mentioned rare condition), it will just re-use them as usual and not store new data chunks. > > If you want to avoid unnecessary chunking, just create or touch a small or empty file in your backup source file set (so that one has the latest mtime, not your 50GB VM disk image) and, if you do snapshots, do the snapshot after that. > > Since only the files cache is used in the display of files status, those files are reported as being added when, really, chunks are already used. From ndbecker2 at gmail.com Thu Dec 20 08:11:07 2018 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 20 Dec 2018 08:11:07 -0500 Subject: [Borgbackup] borg config error Message-ID: Just trying out borg config command, and I get an error: BACKUP/nbecker2 is the location of a (local) repo borg config BACKUP/nbecker2 Local Exception Traceback (most recent call last): File "/home/nbecker/.local/lib/python3.7/site-packages/borg/archiver.py", line 4436, in main exit_code = archiver.run(args) File "/home/nbecker/.local/lib/python3.7/site-packages/borg/archiver.py", line 4368, in run return set_ec(func(args)) File "/home/nbecker/.local/lib/python3.7/site-packages/borg/archiver.py", line 152, in wrapper return method(self, args, repository=repository, **kwargs) File "/home/nbecker/.local/lib/python3.7/site-packages/borg/archiver.py", line 1811, in do_config section, name = args.name.split('.') AttributeError: 'NoneType' object has no attribute 'split' Platform: Linux nbecker7 4.19.9-300.fc29.x86_64 #1 SMP Thu Dec 13 17:25:01 UTC 2018 x86_64 Linux: Fedora 29 Twenty Nine Borg: 1.1.8 Python: CPython 3.7.1 PID: 7987 CWD: /home/nbecker sys.argv: ['/home/nbecker/.local/bin/borg', 'config', 'BACKUP/nbecker2'] SSH_ORIGINAL_COMMAND: None -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Tue Dec 25 16:08:07 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 25 Dec 2018 22:08:07 +0100 Subject: [Borgbackup] borgbackup at 35c3 Message-ID: <23a4dd1c-181b-3736-0279-40cc42996ae7@waldmann-edv.de> Hi, soon, 35c3 (the 35th Chaos Communication Congress) will start in Leipzig, Germany. I'll be there and there will be: - a short "BorgBackup Update" Lightning Talk - a longer "BorgBackup Meetup" self-organized session For more information and updates see there: https://github.com/borgbackup/borg/issues/4224 CU soon, Thomas From devzero at web.de Fri Dec 28 09:08:36 2018 From: devzero at web.de (devzero at web.de) Date: Fri, 28 Dec 2018 15:08:36 +0100 Subject: [Borgbackup] move borg repo including cache to new machine? Message-ID: Hello, i need to move a larger borg-repo to a new machine. apparently, after moving everything (cache dir, repo-dir and staging-dir which is rsynced) , borg is re-chunking everything and creating a new cache for existing source location. does somebody know why and how to avoid this? regards roland From tw at waldmann-edv.de Fri Dec 28 11:28:24 2018 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 28 Dec 2018 17:28:24 +0100 Subject: [Borgbackup] move borg repo including cache to new machine? In-Reply-To: References: Message-ID: <608258ad-3e94-89c9-cb4e-8bacecb81948@waldmann-edv.de> > i need to move a larger borg-repo to a new machine. > > apparently, after moving everything (cache dir, repo-dir and staging-dir which is rsynced) , borg is re-chunking everything and creating a new cache for existing source location. > > does somebody know why and how to avoid this? If you moved the repo "as is" and everything else stayed "as is", it should not do that. Not sure why you mention the cache? It is client-side, so if you only moved the repo, the client side cache does not need moving at all. Also, what do you mean with staging-dir? From devzero at web.de Fri Dec 28 12:31:26 2018 From: devzero at web.de (devzero at web.de) Date: Fri, 28 Dec 2018 18:31:26 +0100 Subject: [Borgbackup] move borg repo including cache to new machine? In-Reply-To: <608258ad-3e94-89c9-cb4e-8bacecb81948@waldmann-edv.de> References: <608258ad-3e94-89c9-cb4e-8bacecb81948@waldmann-edv.de> Message-ID: > Not sure why you mention the cache? It is client-side, so if you only > moved the repo, the client side cache does not need moving at all. for now, we do not have any borg installation remote, we do everything borg related on the backup server only/itself. we first pull everthing from the remote servers via rsync to a local staging dir and then let borg run across that staging area afterwards. sure, we need extra space, but what we want is rotating, space saving, deduplicated "incremental backup forever". so having a rsync copy + borg repo for each of our backed up servers is ok and worth the extra diskspace... regards roland > Gesendet: Freitag, 28. Dezember 2018 um 17:28 Uhr > Von: "Thomas Waldmann" > An: borgbackup at python.org > Betreff: Re: [Borgbackup] move borg repo including cache to new machine? > > > i need to move a larger borg-repo to a new machine. > > > > apparently, after moving everything (cache dir, repo-dir and staging-dir which is rsynced) , borg is re-chunking everything and creating a new cache for existing source location. > > > > does somebody know why and how to avoid this? > > If you moved the repo "as is" and everything else stayed "as is", it > should not do that. > > Not sure why you mention the cache? It is client-side, so if you only > moved the repo, the client side cache does not need moving at all. > > Also, what do you mean with staging-dir? > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From devzero at web.de Fri Dec 28 13:28:40 2018 From: devzero at web.de (devzero at web.de) Date: Fri, 28 Dec 2018 19:28:40 +0100 Subject: [Borgbackup] move borg repo including cache to new machine? Message-ID: apparently, i found it was a serious bug in our backup script. sorry for the noise! > Not sure why you mention the cache? It is client-side, so if you only > moved the repo, the client side cache does not need moving at all. for now, we do not have any borg installation remote, we do everything borg related on the backup server only/itself. we first pull everthing from the remote servers via rsync to a local staging dir and then let borg run across that staging area afterwards. sure, we need extra space, but what we want is rotating, space saving, deduplicated "incremental backup forever". so having a rsync copy + borg repo for each of our backed up servers is ok and worth the extra diskspace... regards roland > Gesendet: Freitag, 28. Dezember 2018 um 17:28 Uhr > Von: "Thomas Waldmann" > An: borgbackup at python.org > Betreff: Re: [Borgbackup] move borg repo including cache to new machine? > > > i need to move a larger borg-repo to a new machine. > > > > apparently, after moving everything (cache dir, repo-dir and staging-dir which is rsynced) , borg is re-chunking everything and creating a new cache for existing source location. > > > > does somebody know why and how to avoid this? > > If you moved the repo "as is" and everything else stayed "as is", it > should not do that. > > Not sure why you mention the cache? It is client-side, so if you only > moved the repo, the client side cache does not need moving at all. > > Also, what do you mean with staging-dir? > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From devzero at web.de Mon Dec 31 04:13:06 2018 From: devzero at web.de (devzero at web.de) Date: Mon, 31 Dec 2018 10:13:06 +0100 Subject: [Borgbackup] move borg repo including cache to new machine? Message-ID: to catch up with this: the issue was not caused by the bug i found, apparently borg detects file change by ctime, not by mtime. so, that explains everything. when i moved everything (ok, move is not exact here - because i did a copy with rsync) to the new machine (borg-repo, rsync staging-dir, borg-chache/security info) the ctime of every file in staging dir is reset (as ctime cannot be transferred) and so borg will rechunk all of them and also effectivly will rebuild chache... so apparently there is no other solution besides permanently switching to --files-cache=mtime or --use-mtime !? regards roland apparently, i found it was a serious bug in our backup script. sorry for the noise! > Not sure why you mention the cache? It is client-side, so if you only > moved the repo, the client side cache does not need moving at all. for now, we do not have any borg installation remote, we do everything borg related on the backup server only/itself. we first pull everthing from the remote servers via rsync to a local staging dir and then let borg run across that staging area afterwards. sure, we need extra space, but what we want is rotating, space saving, deduplicated "incremental backup forever". so having a rsync copy + borg repo for each of our backed up servers is ok and worth the extra diskspace... regards roland > Gesendet: Freitag, 28. Dezember 2018 um 17:28 Uhr > Von: "Thomas Waldmann" > An: borgbackup at python.org > Betreff: Re: [Borgbackup] move borg repo including cache to new machine? > > > i need to move a larger borg-repo to a new machine. > > > > apparently, after moving everything (cache dir, repo-dir and staging-dir which is rsynced) , borg is re-chunking everything and creating a new cache for existing source location. > > > > does somebody know why and how to avoid this? > > If you moved the repo "as is" and everything else stayed "as is", it > should not do that. > > Not sure why you mention the cache? It is client-side, so if you only > moved the repo, the client side cache does not need moving at all. > > Also, what do you mean with staging-dir? > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From ypm.tmp at protonmail.com Mon Dec 31 12:22:50 2018 From: ypm.tmp at protonmail.com (ypm.tmp) Date: Mon, 31 Dec 2018 17:22:50 +0000 Subject: [Borgbackup] 'Multiple repos with the same ID' error Message-ID: [Note that English is not my native language, please excuse any typing errors and awful phrasing] Hi beautiful people! My name is ypm I've been using borg for some months now and I love how all the features it has makes it my ideal backup solution so far: incremental, compressed and encrypted backups <3 Last week I've been having issues with it with my normal workflow. I would do the same thing and it would backup normally, but next time I would connect the drive, it would return the following error, with my usual 'borg create' command, as well as with 'borg list': ??? Cache, or information obtained from the security directory is newer than repository - this is either an attack or unsafe ??? (multiple repos with same ID) Rendering the backup pretty much unusable, so I can't access my data even if I have my passwords, and I don't know what should my next course of action should be in order to both access my data and prevent this error from happening in the future. I'm familiar with some borg commands and performing simple tasks, but not with every single one of them. I can provide more details if any is needed. Thanks in advance and happy holidays! Regards, ypm Details ******* borg version:????? 1.1.8