From avkaplmkt at gmail.com  Thu Apr  2 15:03:18 2020
From: avkaplmkt at gmail.com (Advrk Aplmrkt)
Date: Thu, 2 Apr 2020 20:03:18 +0100
Subject: [Borgbackup] Is it possible to encrypt and existing unencrypted
 borg repository?
Message-ID: <CAN+QS0LwwK1W-cZEmophT6Qv1HPAR09fCr+J42YQ508_X0o0mA@mail.gmail.com>

Hello,

I have an old borg repository (last used with version 1.1.10) that
doesn't have encryption. I would like to keep the archives in this
repository, but I'd prefer to "upgrade" the repository so that is uses
encryption. Can this be done? If so, how? If not, is there anyway to
migrate the data in this repository to a new one that has encryption?

>From what I can tell, encryption is applied at the very beginning when
using the `borg init` command. But is there a way around this? Sorry
if I missed something obvious in the documentation, if so please
correct me. Thank you very much.

From lazyvirus at gmx.com  Thu Apr  2 15:13:04 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Thu, 2 Apr 2020 21:13:04 +0200
Subject: [Borgbackup] Is it possible to encrypt and existing unencrypted
 borg repository?
In-Reply-To: <CAN+QS0LwwK1W-cZEmophT6Qv1HPAR09fCr+J42YQ508_X0o0mA@mail.gmail.com>
References: <CAN+QS0LwwK1W-cZEmophT6Qv1HPAR09fCr+J42YQ508_X0o0mA@mail.gmail.com>
Message-ID: <20200402211304.2c43525b@msi.defcon1.lan>

On Thu, 2 Apr 2020 20:03:18 +0100
Advrk Aplmrkt <avkaplmkt at gmail.com> wrote:

> Hello,

Hi

> I have an old borg repository (last used with version 1.1.10) that
> doesn't have encryption. I would like to keep the archives in this
> repository, but I'd prefer to "upgrade" the repository so that is uses
> encryption. Can this be done? If so, how? If not, is there anyway to
> migrate the data in this repository to a new one that has encryption?
>
> From what I can tell, encryption is applied at the very beginning when
> using the `borg init` command. But is there a way around this? Sorry
> if I missed something obvious in the documentation, if so please
> correct me. Thank you very much.

I'd say the fastest would be:

* create a new & encrypted repo

* borg mount old_backup::backup#1 /somewhere

* use mc to copy from unencrypted mount to new repo

* borg mount new_backup::backup#1 /other_place

* randomly test if everything's right
	cmp /somewhere/path/file1 /other_place/path/file1

* iterate N times where N=Nb_backups

All operations one the server machine, of course.

Jean-Yves

From avkaplmkt at gmail.com  Fri Apr  3 15:15:01 2020
From: avkaplmkt at gmail.com (Advrk Aplmrkt)
Date: Fri, 3 Apr 2020 20:15:01 +0100
Subject: [Borgbackup] Is it possible to encrypt and existing unencrypted
 borg repository?
In-Reply-To: <20200402211304.2c43525b@msi.defcon1.lan>
References: <CAN+QS0LwwK1W-cZEmophT6Qv1HPAR09fCr+J42YQ508_X0o0mA@mail.gmail.com>
 <20200402211304.2c43525b@msi.defcon1.lan>
Message-ID: <CAN+QS0LvhsGD0jCqL7XifA7k2CYUhr_woYkUBTVLKKboMR+d5A@mail.gmail.com>

Thank you!

Can you clarify the step " use mc to copy from unencrypted mount to new repo"?

I can't find the `mc` command in my terminal (would `cp` work?). Also,
did you mean I mount the old backup somewhere, and ask borg to back up
the mounted files into the new, encrypted repo?

If so, this will probably work to achieve most of my goals, but doing
it this way means the *history* of all of my backups will not be
migrated unless I manually mount each backup in the old repository and
back it up into the new one? Sounds tedious considering I have a lot
of backups in the old repo!

On 02/04/2020, Bzzzz <lazyvirus at gmx.com> wrote:
> On Thu, 2 Apr 2020 20:03:18 +0100
> Advrk Aplmrkt <avkaplmkt at gmail.com> wrote:
>
>> Hello,
>
> Hi
>
>> I have an old borg repository (last used with version 1.1.10) that
>> doesn't have encryption. I would like to keep the archives in this
>> repository, but I'd prefer to "upgrade" the repository so that is uses
>> encryption. Can this be done? If so, how? If not, is there anyway to
>> migrate the data in this repository to a new one that has encryption?
>>
>> From what I can tell, encryption is applied at the very beginning when
>> using the `borg init` command. But is there a way around this? Sorry
>> if I missed something obvious in the documentation, if so please
>> correct me. Thank you very much.
>
> I'd say the fastest would be:
>
> * create a new & encrypted repo
>
> * borg mount old_backup::backup#1 /somewhere
>
> * use mc to copy from unencrypted mount to new repo
>
> * borg mount new_backup::backup#1 /other_place
>
> * randomly test if everything's right
> 	cmp /somewhere/path/file1 /other_place/path/file1
>
> * iterate N times where N=Nb_backups
>
> All operations one the server machine, of course.
>
> Jean-Yves

From lazyvirus at gmx.com  Fri Apr  3 15:52:52 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Fri, 3 Apr 2020 21:52:52 +0200
Subject: [Borgbackup] Is it possible to encrypt and existing unencrypted
 borg repository?
In-Reply-To: <CAN+QS0LvhsGD0jCqL7XifA7k2CYUhr_woYkUBTVLKKboMR+d5A@mail.gmail.com>
References: <CAN+QS0LwwK1W-cZEmophT6Qv1HPAR09fCr+J42YQ508_X0o0mA@mail.gmail.com>
 <20200402211304.2c43525b@msi.defcon1.lan>
 <CAN+QS0LvhsGD0jCqL7XifA7k2CYUhr_woYkUBTVLKKboMR+d5A@mail.gmail.com>
Message-ID: <20200403215252.5bdc72ce@msi.defcon1.lan>

On Fri, 3 Apr 2020 20:15:01 +0100
Advrk Aplmrkt <avkaplmkt at gmail.com> wrote:

[I put it back in the loop, as "reply to ML" only took your own e-mail
address in account :(]

> Thank you!
> 
> Can you clarify the step " use mc to copy from unencrypted mount to
> new repo"?
> 
> I can't find the `mc` command in my terminal (would `cp` work?).  

! I made a mistake, you don't copy, you have to use the BB client to
read the mounted old backup in order to make a new backup, encrypted
this time !

> Also,
> did you mean I mount the old backup somewhere, and ask borg to back up
> the mounted files into the new, encrypted repo?  

Yep


> If so, this will probably work to achieve most of my goals, but doing
> it this way means the *history* of all of my backups will not be
> migrated unless I manually mount each backup in the old repository and
> back it up into the new one? Sounds tedious considering I have a lot
> of backups in the old repo!  

Actually, I don't see any other way to do that other than the one I told
you about (erratum included !) - if there is/are really other ways, may
be some others will kick in !?

On the other hand, as you can list old backups, it is trivial to build a
script that'll automatize the whole process, write it, check it, fire it
and do something else (drinking a beer while eating a pizza on top of
your keyboard is a must for somebody working in IT or you can scratch you
legs to get red stockings??)

Oh, and may be you do not need to keep so many backups, I mean if you
have more than one old backup per machine, may be keeping only the last
one would do, think twice !

Jean-Yves

> On 02/04/2020, Bzzzz <lazyvirus at gmx.com> wrote:  
> > On Thu, 2 Apr 2020 20:03:18 +0100
> > Advrk Aplmrkt <avkaplmkt at gmail.com> wrote:
> >  
> >> Hello,  
> >
> > Hi
> >  
> >> I have an old borg repository (last used with version 1.1.10) that
> >> doesn't have encryption. I would like to keep the archives in this
> >> repository, but I'd prefer to "upgrade" the repository so that is
> >> uses encryption. Can this be done? If so, how? If not, is there
> >> anyway to migrate the data in this repository to a new one that has
> >> encryption?
> >>
> >> From what I can tell, encryption is applied at the very beginning
> >> when using the `borg init` command. But is there a way around this?
> >> Sorry if I missed something obvious in the documentation, if so
> >> please correct me. Thank you very much.  
> >
> > I'd say the fastest would be:
> >
> > * create a new & encrypted repo
> >
> > * borg mount old_backup::backup#1 /somewhere
> >
> > * use mc to copy from unencrypted mount to new repo
> >
> > * borg mount new_backup::backup#1 /other_place
> >
> > * randomly test if everything's right
> > 	cmp /somewhere/path/file1 /other_place/path/file1
> >
> > * iterate N times where N=Nb_backups
> >
> > All operations one the server machine, of course.
> >
> > Jean-Yves  


From avkaplmkt at gmail.com  Sun Apr  5 09:39:57 2020
From: avkaplmkt at gmail.com (Advrk Aplmrkt)
Date: Sun, 5 Apr 2020 14:39:57 +0100
Subject: [Borgbackup] Is it possible to encrypt and existing unencrypted
 borg repository?
In-Reply-To: <20200403215252.5bdc72ce@msi.defcon1.lan>
References: <CAN+QS0LwwK1W-cZEmophT6Qv1HPAR09fCr+J42YQ508_X0o0mA@mail.gmail.com>
 <20200402211304.2c43525b@msi.defcon1.lan>
 <CAN+QS0LvhsGD0jCqL7XifA7k2CYUhr_woYkUBTVLKKboMR+d5A@mail.gmail.com>
 <20200403215252.5bdc72ce@msi.defcon1.lan>
Message-ID: <CAN+QS0KC3yqr42b_dOhvHCUva4PLQ+mXCReAFk1yVzHss982Ww@mail.gmail.com>

Thank you for clarifying. I will give this solution a try. In the
meantime if there are other solutions please chime in!

@Gerrit: Oh yeah could `mc` be Midnight Commander? But why?

On 03/04/2020, Bzzzz <lazyvirus at gmx.com> wrote:
> On Fri, 3 Apr 2020 20:15:01 +0100
> Advrk Aplmrkt <avkaplmkt at gmail.com> wrote:
>
> [I put it back in the loop, as "reply to ML" only took your own e-mail
> address in account :(]
>
>> Thank you!
>>
>> Can you clarify the step " use mc to copy from unencrypted mount to
>> new repo"?
>>
>> I can't find the `mc` command in my terminal (would `cp` work?).
>
> ! I made a mistake, you don't copy, you have to use the BB client to
> read the mounted old backup in order to make a new backup, encrypted
> this time !
>
>> Also,
>> did you mean I mount the old backup somewhere, and ask borg to back up
>> the mounted files into the new, encrypted repo?
>
> Yep
>
>
>> If so, this will probably work to achieve most of my goals, but doing
>> it this way means the *history* of all of my backups will not be
>> migrated unless I manually mount each backup in the old repository and
>> back it up into the new one? Sounds tedious considering I have a lot
>> of backups in the old repo!
>
> Actually, I don't see any other way to do that other than the one I told
> you about (erratum included !) - if there is/are really other ways, may
> be some others will kick in !?
>
> On the other hand, as you can list old backups, it is trivial to build a
> script that'll automatize the whole process, write it, check it, fire it
> and do something else (drinking a beer while eating a pizza on top of
> your keyboard is a must for somebody working in IT or you can scratch you
> legs to get red stockings ?)
>
> Oh, and may be you do not need to keep so many backups, I mean if you
> have more than one old backup per machine, may be keeping only the last
> one would do, think twice !
>
> Jean-Yves
>
>> On 02/04/2020, Bzzzz <lazyvirus at gmx.com> wrote:
>> > On Thu, 2 Apr 2020 20:03:18 +0100
>> > Advrk Aplmrkt <avkaplmkt at gmail.com> wrote:
>> >
>> >> Hello,
>> >
>> > Hi
>> >
>> >> I have an old borg repository (last used with version 1.1.10) that
>> >> doesn't have encryption. I would like to keep the archives in this
>> >> repository, but I'd prefer to "upgrade" the repository so that is
>> >> uses encryption. Can this be done? If so, how? If not, is there
>> >> anyway to migrate the data in this repository to a new one that has
>> >> encryption?
>> >>
>> >> From what I can tell, encryption is applied at the very beginning
>> >> when using the `borg init` command. But is there a way around this?
>> >> Sorry if I missed something obvious in the documentation, if so
>> >> please correct me. Thank you very much.
>> >
>> > I'd say the fastest would be:
>> >
>> > * create a new & encrypted repo
>> >
>> > * borg mount old_backup::backup#1 /somewhere
>> >
>> > * use mc to copy from unencrypted mount to new repo
>> >
>> > * borg mount new_backup::backup#1 /other_place
>> >
>> > * randomly test if everything's right
>> > 	cmp /somewhere/path/file1 /other_place/path/file1
>> >
>> > * iterate N times where N=Nb_backups
>> >
>> > All operations one the server machine, of course.
>> >
>> > Jean-Yves

From joehesse at gmail.com  Thu Apr  9 23:06:54 2020
From: joehesse at gmail.com (Joseph Hesse)
Date: Thu, 9 Apr 2020 22:06:54 -0500
Subject: [Borgbackup] Error, "Transport endpoint is not connected"
Message-ID: <99f12516-bf96-e59d-fb2b-7064a5781045@gmail.com>

Hi,

My old computer crashed and I bought a new one.? My backups were to an 
external drive and were always my home directory excluding the .dot 
files.? I did the following.

I fusemounted my last backup and copied it (cp -a -r) to my home directory.

I got the following error - "Transport endpoint is not connected."

I did a repository check and got lots of errors - sorry I panicked and 
don't recall the error messages, this is my only backup.

I did a repository check again and there were no errors.

I tried again to copy the fusemounted backup to my home directory.? The 
following are the errors.

$ cp -a -r * ~
cp: error reading 'DocsPersonal/EvolutionAndIngridBackup/ingrid.tar.gz': 
Software caused connection abort
cp: ?DocsPersonal/EvolutionAndIngridBackup?: Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/Friends': Transport endpoint is not connected
cp: cannot stat 'DocsPersonal/GeneralDocs': Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/OldCalendars': Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/OldLetter': Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/Passport': Transport endpoint is not connected
cp: cannot stat 'DocsPersonal/Polygraph': Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/RedHat': Transport endpoint is not connected
cp: cannot stat 'DocsPersonal/Resume': Transport endpoint is not connected
cp: cannot stat 'DocsPersonal/TechnicalColleges': Transport endpoint is 
not connected
cp: cannot stat 'DocsPersonal/TextFiles': Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/Miscellaneous1': Transport endpoint is not 
connected
cp: cannot stat 'DocsPersonal/Miscellaneous2': Transport endpoint is not 
connected

+ 20 more lines

Before my old computer crashed I did a repository check and got no errors.

Is there a way I can do a complete restore?

Thank you,

Joe




From gait at atcomputing.nl  Fri Apr 10 03:16:18 2020
From: gait at atcomputing.nl (Gerrit A. Smit :: AT)
Date: Fri, 10 Apr 2020 09:16:18 +0200
Subject: [Borgbackup] Error, "Transport endpoint is not connected"
In-Reply-To: <99f12516-bf96-e59d-fb2b-7064a5781045@gmail.com>
References: <99f12516-bf96-e59d-fb2b-7064a5781045@gmail.com>
Message-ID: <CAF4+MbnbPz0ZOFsn0RdJv3aRTXA=GWF3ogZ2hL8W_PydyyyFtw@mail.gmail.com>

Hello,

Did you try to install the BorgBackup binary?
I had the same problem (endpoint not connected) a few days ago which went
away after using the binary only install.


-- 
AT COMPUTING BV
Beheer Technische Infrastructuur

Gerrit A. Smit
+31 24 3527 222
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200410/5cae0cf9/attachment.html>

From tw at waldmann-edv.de  Fri Apr 10 08:51:21 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 10 Apr 2020 14:51:21 +0200
Subject: [Borgbackup] Error, "Transport endpoint is not connected"
In-Reply-To: <99f12516-bf96-e59d-fb2b-7064a5781045@gmail.com>
References: <99f12516-bf96-e59d-fb2b-7064a5781045@gmail.com>
Message-ID: <26f8247f-a2ef-4ae1-45f2-5ac6ff2095da@waldmann-edv.de>

> I fusemounted my last backup and copied it (cp -a -r) to my home directory.

You usually rather want rsync -aH (-v)? (--numeric-ids)? ...


> I got the following error - "Transport endpoint is not connected."

This happens if the borg mount process dies.

Use "borg mount -f" to see why it is dying.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From mail at tgries.de  Sat Apr 11 17:30:56 2020
From: mail at tgries.de (Thomas Gries)
Date: Sat, 11 Apr 2020 23:30:56 +0200
Subject: [Borgbackup] borg list -- list only files matching a pattern - two
 questions
Message-ID: <5a58bb67-d7c4-e6ca-dca7-b38862656c27@tgries.de>

https://borgbackup.readthedocs.io/en/stable/usage/list.html says:

    borg [common options] list [options] [REPOSITORY_OR_ARCHIVE] [PATH...]
    ...PATH ??? paths to list;_/*patterns are supported (*)*/_


  * Problem:
    I want to list all files/paths in my repo::archive with a certain
    string example: "_/*music*/_" in the path.


This command does not give any result:

  * borg list repo::archive *music*


And this command does not limit as wanted:

  * borg list --pattern="+*music*" repo::archive


So at the moment I only see a chance to list all files and to pipe them
to a grep like in:

  * borg list repo::archive | grep -i music


Question 1:
What is the correct "borg list" command to list only files matching a
certain expression?

Question 2:
What exactly do you mean by "patterns are supported" in your help text (*) ?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200411/a018eb2f/attachment.html>

From l0f4r0 at tuta.io  Sun Apr 12 04:29:14 2020
From: l0f4r0 at tuta.io (l0f4r0 at tuta.io)
Date: Sun, 12 Apr 2020 10:29:14 +0200 (CEST)
Subject: [Borgbackup] borg list -- list only files matching a pattern -
 two questions
In-Reply-To: <5a58bb67-d7c4-e6ca-dca7-b38862656c27@tgries.de>
References: <5a58bb67-d7c4-e6ca-dca7-b38862656c27@tgries.de>
Message-ID: <M4hU8sJ--3-2@tuta.io>

Hi,

I think "borg help patterns" is all you need :)

Best regards,
l0f4r0

From l0f4r0 at tuta.io  Sun Apr 12 07:16:35 2020
From: l0f4r0 at tuta.io (l0f4r0 at tuta.io)
Date: Sun, 12 Apr 2020 13:16:35 +0200 (CEST)
Subject: [Borgbackup] borg list -- list only files matching a pattern -
 two questions
In-Reply-To: <cf5da0d5-d0a5-22ec-5d21-8aead90a98ed@tgries.de>
References: <5a58bb67-d7c4-e6ca-dca7-b38862656c27@tgries.de>
 <M4hU8sJ--3-2@tuta.io> <cf5da0d5-d0a5-22ec-5d21-8aead90a98ed@tgries.de>
Message-ID: <M4i5k2m--3-2@tuta.io>

Hi again,

12 avr. 2020 ? 11:57 de mail at tgries.de:

> Am 12.04.20 um 10:29 schrieb l0f4r0--- via Borgbackup:
>
>> I think "borg help patterns" is all you need :)
>>
> No. As explained, this is of no help, as it does not solve the problem.
>
For example, have you tried the following possibility according to the documentation?
borg list repo::archive "re:music"

> It is simply not working.
>
It works for me...

Regards,
l0f4r0

From mail at tgries.de  Sun Apr 12 08:13:01 2020
From: mail at tgries.de (Thomas Gries)
Date: Sun, 12 Apr 2020 14:13:01 +0200
Subject: [Borgbackup] borg list -- list only files matching a pattern -
 two questions [SOLVED]
In-Reply-To: <M4i5k2m--3-2@tuta.io>
References: <5a58bb67-d7c4-e6ca-dca7-b38862656c27@tgries.de>
 <M4hU8sJ--3-2@tuta.io> <cf5da0d5-d0a5-22ec-5d21-8aead90a98ed@tgries.de>
 <M4i5k2m--3-2@tuta.io>
Message-ID: <d1750d2f-bed1-36ff-e874-1ebc224fd0e1@tgries.de>

For example, have you tried the following possibility according to the
documentation?
> borg list repo::archive "re:music"
>
Okay, that worked. indeed.

The reason why it did not work (for me) is that I understood from your
documentation that "pattern" can only be an argument after "--pattern"
or "--exclude",
But this appeared to be a wrong view, as I now learnt.


I will make - later - a reasoned pull request to add such an example to
the documentation.


Another aspect:
The entry "pattern" should also get an distinct entry in the
documentation so that it can be found when searching for it.
Currently it cannot be found quickly when you do not know where to look
it up.


*_Issue solved._*

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200412/fee113c0/attachment.html>

From mail at tgries.de  Mon Apr 13 16:06:34 2020
From: mail at tgries.de (Thomas Gries)
Date: Mon, 13 Apr 2020 22:06:34 +0200
Subject: [Borgbackup] Where can I file change requests for
 https://borgbackup.readthedocs.io ?
Message-ID: <06f7069e-b22d-60b1-fc94-009b63f8ebf1@tgries.de>

I suggest to add something and to change something on (example)

https://borgbackup.readthedocs.io/en/stable/usage/help.html
https://borgbackup.readthedocs.io/en/master/usage/help.html

but I do not know where and how exactly I can submit a change request.

Please can someone explain the correct way?



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200413/79396cfd/attachment.html>

From public at enkore.de  Mon Apr 13 16:32:57 2020
From: public at enkore.de (Marian Beermann)
Date: Mon, 13 Apr 2020 22:32:57 +0200
Subject: [Borgbackup] Where can I file change requests for
 https://borgbackup.readthedocs.io ?
In-Reply-To: <06f7069e-b22d-60b1-fc94-009b63f8ebf1@tgries.de>
References: <06f7069e-b22d-60b1-fc94-009b63f8ebf1@tgries.de>
Message-ID: <f14741a6-aef8-6b37-d881-03354a2e03b6@enkore.de>

Hi,

The usage docs are built from the command line parser help coded in
src/borg/archiver.py (see
<https://borgbackup.readthedocs.io/en/stable/development.html#documentation>)
for details.

All other docs can be edited directly in the docs/ folder. The git
repository lives at <https://github.com/borgbackup/borg/>.

-Marian

Am 13.04.20 um 22:06 schrieb Thomas Gries:
> I suggest to add something and to change something on (example)
> 
> https://borgbackup.readthedocs.io/en/stable/usage/help.html
> https://borgbackup.readthedocs.io/en/master/usage/help.html
> 
> but I do not know where and how exactly I can submit a change request.
> 
> Please can someone explain the correct way?
> 
> 
> 
> 
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
> 


From mail at tgries.de  Mon Apr 13 20:09:37 2020
From: mail at tgries.de (Thomas Gries)
Date: Tue, 14 Apr 2020 02:09:37 +0200
Subject: [Borgbackup] Suggestion to improve the documentation regarding the
 repo encryption
Message-ID: <036a93b9-1e6d-9315-0a79-981bb42923d9@tgries.de>

Re: https://borgbackup.readthedocs.io/en/stable/usage/key.html
and
https://borgbackup.readthedocs.io/en/stable/quickstart.html#repository-encryption


As far as I understand how borg's encryption works [with a keyfile], one
can change its /passphrase/ but can not remove or change the encryption
as such.
I also? suppose that it is not possible to change a repo from
"non-encrypted" to "encrypted".

[Please correct me if I am wrong.]


I'd simply like to suggest to improve our Borg FAQ section with two
question sections [and answers] like

  * "Question 1: When I know the passphrase, can I remove the encryption
    of the repo?"
  * "Question 2: Can I change a repo from 'non-encrypted' to 'encrypted'?"


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200414/d51bd3e4/attachment.html>

From l0f4r0 at tuta.io  Tue Apr 14 15:19:55 2020
From: l0f4r0 at tuta.io (l0f4r0 at tuta.io)
Date: Tue, 14 Apr 2020 21:19:55 +0200 (CEST)
Subject: [Borgbackup] Suggestion to improve the documentation regarding
 the repo encryption
In-Reply-To: <036a93b9-1e6d-9315-0a79-981bb42923d9@tgries.de>
References: <036a93b9-1e6d-9315-0a79-981bb42923d9@tgries.de>
Message-ID: <M4u7YPn--3-2@tuta.io>

Hi,

14 avr. 2020 ? 02:09 de mail at tgries.de:

> As far as I understand how borg's encryption works [with a      keyfile], one can change its > passphrase>  but can not remove      or change the encryption as such.
>  I also? suppose that it is not possible to change a repo from      "non-encrypted" to "encrypted".
>  
>  [Please correct me if I am wrong.]
>
Indeed, this is my understanding as well.
See?https://borgbackup.readthedocs.io/en/stable/usage/init.html: "Encryption can be enabled at repository init time. It cannot be changed later.".

Best regards,
l0f4r0

From l0f4r0 at tuta.io  Tue Apr 14 15:36:35 2020
From: l0f4r0 at tuta.io (l0f4r0 at tuta.io)
Date: Tue, 14 Apr 2020 21:36:35 +0200 (CEST)
Subject: [Borgbackup] borg list -- list only files matching a pattern -
 two questions [SOLVED]
In-Reply-To: <d1750d2f-bed1-36ff-e874-1ebc224fd0e1@tgries.de>
References: <5a58bb67-d7c4-e6ca-dca7-b38862656c27@tgries.de>
 <M4hU8sJ--3-2@tuta.io> <cf5da0d5-d0a5-22ec-5d21-8aead90a98ed@tgries.de>
 <M4i5k2m--3-2@tuta.io> <d1750d2f-bed1-36ff-e874-1ebc224fd0e1@tgries.de>
Message-ID: <M4uBMZu--3-2@tuta.io>

Hi,

12 avr. 2020 ? 14:13 de mail at tgries.de:

>   The reason why it did not work (for me) is that I understood from    your documentation [...]
>
This is not *my* documentation.
I'm just an humble user ;)

>  > Issue solved.
>
I'm glad it is.

Best regards,
l0f4r0

From mail at tgries.de  Sun Apr 19 14:41:41 2020
From: mail at tgries.de (Thomas Gries)
Date: Sun, 19 Apr 2020 20:41:41 +0200
Subject: [Borgbackup] Multi-archive searches in Borg Backup?
Message-ID: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>

tl,dr
Have you ever thought about implementing a native

  * "borg lookup for files in all archives of a repo" command?


------------------------------------------------------------------------
Long version:

I wanted to look up for files, generally matching a regular expression,
in any(all) archives of a repo.

I found this older post:

"Multi-archive searches in Borg Backup?"
https://mail.python.org/pipermail/borgbackup/2016q3/000402.html

which mentions three possibilities

FUSE mounted repositories have a "versions view" where all versions of
a file are available:
- borg mount

- borg-diff and
- borg-list

are the two other possibilities, but these are treating only two
respectively one archive of a repo.

Because as Thomas says? "that fuse support is not present /
unproblematic on all platforms.", the "borg mount" may not a generally
available method:
I bring up the question therefore:
Have you ever thought about implementing a native "borg lookup for files
in all archives of a repo" command?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200419/548e7d26/attachment.html>

From mail at tgries.de  Sun Apr 19 15:05:29 2020
From: mail at tgries.de (Thomas Gries)
Date: Sun, 19 Apr 2020 21:05:29 +0200
Subject: [Borgbackup] Multi-archive searches in Borg Backup?
In-Reply-To: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>
References: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>
Message-ID: <491d2863-a8ec-91b8-4d65-82b6c638dfae@tgries.de>

At the moment, I would suggest this ad-hoc solution

search for $pattern in all archives of $repo

for archive in $(borg list --short $repo);do echo borg list --short
$repo::$archive "re:$pattern";done

> tl,dr
> Have you ever thought about implementing a native
>
>   * "borg lookup for files in all archives of a repo" command?
>
>
> ------------------------------------------------------------------------
> Long version:
>
> I wanted to look up for files, generally matching a regular
> expression, in any(all) archives of a repo.
>
> I found this older post:
>
> "Multi-archive searches in Borg Backup?"
> https://mail.python.org/pipermail/borgbackup/2016q3/000402.html
>
> which mentions three possibilities
>
> FUSE mounted repositories have a "versions view" where all versions of
> a file are available:
> - borg mount
>
> - borg-diff and
> - borg-list
>
> are the two other possibilities, but these are treating only two
> respectively one archive of a repo.
>
> Because as Thomas says? "that fuse support is not present /
> unproblematic on all platforms.", the "borg mount" may not a generally
> available method:
> I bring up the question therefore:
> Have you ever thought about implementing a native "borg lookup for
> files in all archives of a repo" command?
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200419/1bd0590e/attachment.html>

From tw at waldmann-edv.de  Sun Apr 19 15:24:11 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 19 Apr 2020 21:24:11 +0200
Subject: [Borgbackup] Multi-archive searches in Borg Backup?
In-Reply-To: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>
References: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>
Message-ID: <5ec513f6-7fe3-19a2-67a7-5dccf7546cf4@waldmann-edv.de>

> Because as Thomas says? "that fuse support is not present /
> unproblematic on all platforms.", the "borg mount" may not a generally
> available method:
> I bring up the question therefore:
> Have you ever thought about implementing a native "borg lookup for files
> in all archives of a repo" command?

If you do that "ad-hoc", that's rather expensive (slow), especially if
you need multiple runs, because for every run, you'ld have to go through
all archives...

FUSE is usually there on Linux, also on FreeBSD and macOS, but might
need some configuration and docs reading. It's not less expensive than
one run of the above, but at least you only need it once when searching
for stuff.

Another way is to use borg list or borg create --list and dump the
output to files, then search in these files.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From mail at tgries.de  Sun Apr 19 15:37:31 2020
From: mail at tgries.de (Thomas Gries)
Date: Sun, 19 Apr 2020 21:37:31 +0200
Subject: [Borgbackup] Multi-archive searches in Borg Backup?
In-Reply-To: <5ec513f6-7fe3-19a2-67a7-5dccf7546cf4@waldmann-edv.de>
References: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>
 <5ec513f6-7fe3-19a2-67a7-5dccf7546cf4@waldmann-edv.de>
Message-ID: <20a7743f-cd32-5dd7-b135-2a94bdd695ea@tgries.de>

> Another way is to use borg list or borg create --list and dump the
output to files, then search in these files.

As far as I understand, I also have to run n times "borg list" of a
repo, so I need the "do" loop.

Question:

Do you have a better idea than running for all archives "borg list" ?
I could not find an example.

From tw at waldmann-edv.de  Sun Apr 19 16:10:15 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 19 Apr 2020 22:10:15 +0200
Subject: [Borgbackup] Multi-archive searches in Borg Backup?
In-Reply-To: <20a7743f-cd32-5dd7-b135-2a94bdd695ea@tgries.de>
References: <d36afeac-2a29-9ea6-84ad-fe085ed2b01c@tgries.de>
 <5ec513f6-7fe3-19a2-67a7-5dccf7546cf4@waldmann-edv.de>
 <20a7743f-cd32-5dd7-b135-2a94bdd695ea@tgries.de>
Message-ID: <80d019a2-e18f-36e4-ae60-f549e7deda5c@waldmann-edv.de>

> As far as I understand, I also have to run n times "borg list" of a
> repo, so I need the "do" loop.

Sure, you need the for loop.

> Do you have a better idea than running for all archives "borg list" ?
> I could not find an example.

borg mount [-o versions] repo /mnt

and then do anything you like below /mnt

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Wed Apr 22 16:44:08 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Wed, 22 Apr 2020 22:44:08 +0200
Subject: [Borgbackup] borgbackup 1.2.0 alpha 8 released!
Message-ID: <6de513cc-a327-a437-1b5f-3ad58da808e5@waldmann-edv.de>


Released #borgbackup 1.2.0 alpha 8 - not for production, only for testing!

https://github.com/borgbackup/borg/releases/tag/1.2.0a8


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From hpj at urpla.net  Sun Apr 26 07:21:29 2020
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Sun, 26 Apr 2020 13:21:29 +0200
Subject: [Borgbackup] Some tests fail with Python 3.8
Message-ID: <4834494.VV8eii2ndf@xrated>

Hi,

our openSUSE Factory builds are failing since switching to 3.8 (I guess).
It might be related to some other library update.

[  878s] =================================== FAILURES 
===================================
[  878s] ______________________ test_progress_percentage_sameline 
_______________________
[  878s] 
[  878s] capfd = <_pytest.capture.CaptureFixture object at 0x7f1987446220>
[  878s] monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 
0x7f1987445c10>
[  878s] 
[  878s]     def test_progress_percentage_sameline(capfd, monkeypatch):
[  878s]         # run the test as if it was in a 4x1 terminal
[  878s]         monkeypatch.setenv('COLUMNS', '4')
[  878s]         monkeypatch.setenv('LINES', '1')
[  878s]         pi = ProgressIndicatorPercent(1000, step=5, start=0, 
msg="%3.0f%%")
[  878s]         pi.logger.setLevel('INFO')
[  878s]         pi.show(0)
[  878s]         out, err = capfd.readouterr()
[  878s] >       assert err == '  0%\r'
[  878s] E       AssertionError: assert '  0%\n' == '  0%\r'
[  878s] E         -   0%
[  878s] E         ?     ^
[  878s] E         +   0%

[  878s] E         ?     ^
[  878s] 
[  878s] borg/testsuite/helpers.py:748: AssertionError
[  878s] ________________________ test_progress_percentage_step 
_________________________
[  878s] 
[  878s] capfd = <_pytest.capture.CaptureFixture object at 0x7f1990c8d460>
[  878s] monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 
0x7f1987459a90>
[  878s] 
[  878s]     def test_progress_percentage_step(capfd, monkeypatch):
[  878s]         # run the test as if it was in a 4x1 terminal
[  878s]         monkeypatch.setenv('COLUMNS', '4')
[  878s]         monkeypatch.setenv('LINES', '1')
[  878s]         pi = ProgressIndicatorPercent(100, step=2, start=0, 
msg="%3.0f%%")
[  878s]         pi.logger.setLevel('INFO')
[  878s]         pi.show()
[  878s]         out, err = capfd.readouterr()
[  878s] >       assert err == '  0%\r'
[  878s] E       assert '--- Logging ...guments: ()\n' == '  0%\r'
[  878s] E         +   0%

[  878s] E         - --- Logging error ---
[  878s] E         - Traceback (most recent call last):
[  878s] E         -   File "/usr/lib64/python3.8/logging/__init__.py", line 
1084, in emit
[  878s] E         -     stream.write(msg + self.terminator)
[  878s] E         - ValueError: I/O operation on closed file.
[  878s] E         - Call stack:...
[  878s] E         
[  878s] E         ...Full output truncated (79 lines hidden), use '-vv' to 
show
[  878s] 
[  878s] borg/testsuite/helpers.py:769: AssertionError
[  878s] =============================== warnings summary 
===============================
[  878s] /usr/lib/python3.8/site-packages/_pytest/mark/structures.py:323
[  878s]   /usr/lib/python3.8/site-packages/_pytest/mark/structures.py:323: 
PytestUnknownMarkWarning: Unknown pytest.mark.allow_cache_wipe - is this a 
typo?  You can register custom marks to avoid this warning - for details, see 
https://docs.pytest.org/en/latest/mark.html
[  878s]     warnings.warn(
[  878s] 
[  878s] -- Docs: https://docs.pytest.org/en/latest/warnings.html
[  878s] = 2 failed, 1122 passed, 176 skipped, 54 deselected, 2 xfailed, 1 
warning in 748.67s (0:12:28) =
[  878s] error: Bad exit status from /var/tmp/rpm-tmp.vna18f (%check)
[  878s] 

Any suggestions on how to deal with these test failures?

I can disable those tests, but that would be always the last resort.

Thanks,
Pete



From hpj at urpla.net  Sun Apr 26 07:23:04 2020
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Sun, 26 Apr 2020 13:23:04 +0200
Subject: [Borgbackup] Some tests fail with Python 3.8
In-Reply-To: <4834494.VV8eii2ndf@xrated>
References: <4834494.VV8eii2ndf@xrated>
Message-ID: <5001446.3hxpXhLiCH@xrated>

Am Sonntag, 26. April 2020, 13:21:29 CEST schrieb Hans-Peter Jansen:
> Hi,
> 
> our openSUSE Factory builds are failing since switching to 3.8 (I guess).
> It might be related to some other library update.

Prematurely send. :-(

Here are the full build logs:

https://build.opensuse.org/package/show/Archiving:Backup/borgbackup

Cheers,
Pete



From felix.schwarz at oss.schwarz.eu  Sun Apr 26 07:31:34 2020
From: felix.schwarz at oss.schwarz.eu (Felix Schwarz)
Date: Sun, 26 Apr 2020 13:31:34 +0200
Subject: [Borgbackup] Some tests fail with Python 3.8
In-Reply-To: <4834494.VV8eii2ndf@xrated>
References: <4834494.VV8eii2ndf@xrated>
Message-ID: <df554d79-417d-d9ef-4dbf-5468a679f309@oss.schwarz.eu>


Am 26.04.20 um 13:21 schrieb Hans-Peter Jansen:
> our openSUSE Factory builds are failing since switching to 3.8 (I guess).
> It might be related to some other library update.

In general Python 3.8 should be fine. The test suite passes for us in Fedora
32 (which uses 3.8 as well) and even Python 3.9 (Fedora 33) works (IIRC).

Latest Fedora builds for borgbackup 1.1.11:

Fedora 33:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1475335

Fedora 32:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1475336

If you are looking for exact library versions used in Fedora maybe Koschei
could help:
https://koschei.fedoraproject.org/package/borgbackup?collection=f32

Felix

From hpj at urpla.net  Sun Apr 26 13:25:47 2020
From: hpj at urpla.net (Hans-Peter Jansen)
Date: Sun, 26 Apr 2020 19:25:47 +0200
Subject: [Borgbackup] Some tests fail with Python 3.8
In-Reply-To: <df554d79-417d-d9ef-4dbf-5468a679f309@oss.schwarz.eu>
References: <4834494.VV8eii2ndf@xrated>
 <df554d79-417d-d9ef-4dbf-5468a679f309@oss.schwarz.eu>
Message-ID: <3408311.iVCocL2L46@xrated>

Am Sonntag, 26. April 2020, 13:31:34 CEST schrieb Felix Schwarz:
> Am 26.04.20 um 13:21 schrieb Hans-Peter Jansen:
> > our openSUSE Factory builds are failing since switching to 3.8 (I guess).
> > It might be related to some other library update.
> 
> In general Python 3.8 should be fine. The test suite passes for us in Fedora
> 32 (which uses 3.8 as well) and even Python 3.9 (Fedora 33) works (IIRC).
> 
> Latest Fedora builds for borgbackup 1.1.11:
> 
> Fedora 33:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1475335
> 
> Fedora 32:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1475336
> 
> If you are looking for exact library versions used in Fedora maybe Koschei
> could help:
> https://koschei.fedoraproject.org/package/borgbackup?collection=f32

Thanks, Felix, much appreciated. 

Will dive into this tomorrow.

Pete



From gait at atcomputing.nl  Wed Apr 29 14:53:50 2020
From: gait at atcomputing.nl (Gerrit A. Smit :: AT)
Date: Wed, 29 Apr 2020 20:53:50 +0200
Subject: [Borgbackup] Complete download of archive fails - mostly
Message-ID: <CAF4+Mbktx=K5vtcgRbBUk1rbtZ27mvBtsztA2JzGabrQBVSDuQ@mail.gmail.com>

Hello,


I am using rsync.net as a provider of disk-space (very cheap as no
ZFS-snapshots are needed).

After uploading the first 10 archives it was about time to test things the
other way around,
so I did a borg extract ... --dry-run ...

Only to see that at about 10% of the work it stopped working.
SSH-server closed connection.
This was from a Google Compute host.

When I did the same thing from home, with a very poor internet connection,
it took several hours, but eventually all was OK.

I did try using the pv-wrapper-script, which helped only a bit (at most
60%).

What could be wrong 'in the cloud'?

rsync.net support just told me they drastically enlarged my quota, but I
fail to see how that could help: outside of my host, the internet should
only see compressed/encrypted/deduplicated contents, right?


-- 
AT COMPUTING BV
Beheer Technische Infrastructuur

Gerrit A. Smit
+31 24 3527 222
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200429/bcd89cea/attachment.html>

From gait at atcomputing.nl  Wed Apr 29 16:37:12 2020
From: gait at atcomputing.nl (Gerrit A. Smit :: AT)
Date: Wed, 29 Apr 2020 22:37:12 +0200
Subject: [Borgbackup] (How) can I use patterns with borg extract?
Message-ID: <CAF4+Mb=1B37kcktozqdqfq6Yu-a_KEbbnFdkvfcX2ZN5zTyF-A@mail.gmail.com>

Dear list,


How can I use borg extract .... --pattern= ....
to extract only PDF-files?

-- 
AT COMPUTING BV
Beheer Technische Infrastructuur

Gerrit A. Smit
+31 24 3527 222
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200429/84b3c476/attachment.html>

From mrobti at insiberia.net  Sun May  3 17:13:13 2020
From: mrobti at insiberia.net (MRob)
Date: Sun, 03 May 2020 21:13:13 +0000
Subject: [Borgbackup] Storage, CPU, RAM comparisons
Message-ID: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>

Hello,
I reading about borg and looking exciting to me. Thank you for a great 
software!

Can I ask, I have only seen one storage and CPU comparison:
https://github.com/gilbertchen/benchmarking

With deduplication, why is disk usage so much higher than duplicity? Can 
I assume disk usage of duplicity would be similar to rdiff-backup? 
Because rdiff/duplicity stores deltas of each file but deduplication 
strategy stores common file blocks I expect borg to being far smaller 
storage needs. Why is that wrong?

Also like to ask what CPU, RAM usage of borg. I read only one place CPU 
use if very high for borg compare to rsync.
Thank you!

From panayotis at panayotis.com  Sun May  3 17:25:15 2020
From: panayotis at panayotis.com (Panayotis Katsaloulis)
Date: Mon, 4 May 2020 00:25:15 +0300
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
Message-ID: <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>

Just stating my own experience.
In a real-file source-code related of 6GB (all my code I ever written with their datafiles) Borg backup was the one with the best storage handling and at the same time the ability to directly mount your repository and look back in history.
There?s no other tool that can do this best than borgbackup.


--
Panayotis
On 4 May 2020, 12:22 AM +0300, MRob <mrobti at insiberia.net>, wrote:
> Hello,
> I reading about borg and looking exciting to me. Thank you for a great
> software!
>
> Can I ask, I have only seen one storage and CPU comparison:
> https://github.com/gilbertchen/benchmarking
>
> With deduplication, why is disk usage so much higher than duplicity? Can
> I assume disk usage of duplicity would be similar to rdiff-backup?
> Because rdiff/duplicity stores deltas of each file but deduplication
> strategy stores common file blocks I expect borg to being far smaller
> storage needs. Why is that wrong?
>
> Also like to ask what CPU, RAM usage of borg. I read only one place CPU
> use if very high for borg compare to rsync.
> Thank you!
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200504/36c07c3f/attachment.html>

From dastapov at gmail.com  Sun May  3 17:29:34 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Sun, 3 May 2020 22:29:34 +0100
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
Message-ID: <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>

That github project benchmarks attic (borgs predecessor), not borg.

Look inside common.sh for proof.

On Sun, May 3, 2020, 22:25 Panayotis Katsaloulis <panayotis at panayotis.com>
wrote:

> Just stating my own experience.
> In a real-file source-code related of 6GB (all my code I ever written with
> their datafiles) Borg backup was the one with the best storage handling and
> at the same time the ability to directly mount your repository and look
> back in history.
> There?s no other tool that can do this best than borgbackup.
>
>
> --
> Panayotis
> On 4 May 2020, 12:22 AM +0300, MRob <mrobti at insiberia.net>, wrote:
>
> Hello,
> I reading about borg and looking exciting to me. Thank you for a great
> software!
>
> Can I ask, I have only seen one storage and CPU comparison:
> https://github.com/gilbertchen/benchmarking
>
> With deduplication, why is disk usage so much higher than duplicity? Can
> I assume disk usage of duplicity would be similar to rdiff-backup?
> Because rdiff/duplicity stores deltas of each file but deduplication
> strategy stores common file blocks I expect borg to being far smaller
> storage needs. Why is that wrong?
>
> Also like to ask what CPU, RAM usage of borg. I read only one place CPU
> use if very high for borg compare to rsync.
> Thank you!
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200503/f9cc0126/attachment.html>

From mrobti at insiberia.net  Sun May  3 17:54:32 2020
From: mrobti at insiberia.net (MRob)
Date: Sun, 03 May 2020 21:54:32 +0000
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
Message-ID: <898e273b561646e9808f7d060bcca690@insiberia.net>

On 2020-05-03 21:29, Dmitry Astapov wrote:
> That github project benchmarks attic (borgs predecessor), not borg.

Reading the setup info it's BorgBackup 1.1.0b6 and the author confirm 
here:

https://news.ycombinator.com/item?id=14796936

Maybe borg improved from attic, but its not the point. Borg, attic, 
duplicacy based on deduplication using massive higher storage space in 
relation to duplicity (and rdiff-backup?). I dont' understand why 
file-based delta is more storage efficient then deduplication which can 
consolidate chunks from all files in the repo. I expect the opposite 
storage use ratio comparison.

Also anyone can comment borg CPU/RAM use in comparison to other?


> On Sun, May 3, 2020, 22:25 Panayotis Katsaloulis 
> <panayotis at panayotis.com>
> wrote:
> 
>> Just stating my own experience.
>> In a real-file source-code related of 6GB (all my code I ever written 
>> with
>> their datafiles) Borg backup was the one with the best storage 
>> handling and
>> at the same time the ability to directly mount your repository and 
>> look
>> back in history.
>> There?s no other tool that can do this best than borgbackup.
>> 
>> 
>> --
>> Panayotis
>> On 4 May 2020, 12:22 AM +0300, MRob <mrobti at insiberia.net>, wrote:
>> 
>> Hello,
>> I reading about borg and looking exciting to me. Thank you for a great
>> software!
>> 
>> Can I ask, I have only seen one storage and CPU comparison:
>> https://github.com/gilbertchen/benchmarking
>> 
>> With deduplication, why is disk usage so much higher than duplicity? 
>> Can
>> I assume disk usage of duplicity would be similar to rdiff-backup?
>> Because rdiff/duplicity stores deltas of each file but deduplication
>> strategy stores common file blocks I expect borg to being far smaller
>> storage needs. Why is that wrong?
>> 
>> Also like to ask what CPU, RAM usage of borg. I read only one place 
>> CPU
>> use if very high for borg compare to rsync.
>> Thank you!
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
>> 
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
>> 

From devzero at web.de  Mon May  4 04:04:36 2020
From: devzero at web.de (Roland)
Date: Mon, 4 May 2020 10:04:36 +0200
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
Message-ID: <246b5783-eafa-2c34-20f5-f37ca743b5b9@web.de>

 >Although duplicity was the most storage efficient, it should be noted
that it uses zlib, which is known to compress better than lz4 used by
Duplicacy and Attic.

*facepalm*


Am 03.05.20 um 23:13 schrieb MRob:
> Hello,
> I reading about borg and looking exciting to me. Thank you for a great
> software!
>
> Can I ask, I have only seen one storage and CPU comparison:
> https://github.com/gilbertchen/benchmarking
>
> With deduplication, why is disk usage so much higher than duplicity?
> Can I assume disk usage of duplicity would be similar to rdiff-backup?
> Because rdiff/duplicity stores deltas of each file but deduplication
> strategy stores common file blocks I expect borg to being far smaller
> storage needs. Why is that wrong?
>
> Also like to ask what CPU, RAM usage of borg. I read only one place
> CPU use if very high for borg compare to rsync.
> Thank you!
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup

From dastapov at gmail.com  Mon May  4 05:54:11 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Mon, 4 May 2020 10:54:11 +0100
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <898e273b561646e9808f7d060bcca690@insiberia.net>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
Message-ID: <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>

If you clone the repo and try to run the test yourself, you will find that
it tries to run a binary called "attic", not "borg".
So the author chose to rename "borg" to "attic" because ... why exactly?

Granted, it tries to run it with borg command line keys, but overall this
is suspicions and renders test unreproducible out-of-the-box.

As others have stated, comparing different compression modes is also quite
meaningless. For example, after fixing the scripts in the repo in the
minimal way i ran them with lz4 and zstd compression and got:

861M linux-attic-storage.lz4
599M linux-attic-storage.zstd

As you can see, zstd is 69% of lz4, which is not all insignificant. You can
run it with zlib if you are curious.

To answer your original question:

>
> Maybe borg improved from attic, but its not the point. Borg, attic,
> duplicacy based on deduplication using massive higher storage space in
> relation to duplicity (and rdiff-backup?). I dont' understand why
> file-based delta is more storage efficient then deduplication which can
> consolidate chunks from all files in the repo. I expect the opposite
> storage use ratio comparison.
>

Let's remove the first backup of 12 and compare disk usage afterward, shall
we?
Oh, wait, you can't do it with duplicity and rdiff-backup, right? Not
without taking another full backup first.

In real-life scenarios when taking full backups could be prohibitively
expensive (especially scenarios like massive amounts of data over
high-latency low-speed links), rdiff-backup/duplicity approach becomes
simply unviable, disk space saving or not, because you either have to keep
all backups infinitely long (and eventually run out of storage space) or
you need to pay the price of full backup once in a while (which will likely
overshadow all the time/disk space savings you have made previously).

Eventually, implementing meaningful retention strategy with
duplicity/rdiff-backups becomes major pain in the back. Let's say that I
have 2Tb of photos on my computer which I want to backup on an hourly basis
over wifi and retain just a week of hourly backups. With borg, i pay the
price of transeffing all of 2Tb just once, and afterwards my backups are
proportional to the number of photos changed. With duplicity/rdiff-backup I
will have to upload 2Tb over wifi once a week.

So some fraction of extra space taken by bog will be manifests that tie
blocks in the storage to files that use them.

Source: have been rdiff-backup and duplicity user for a while before moving
to attic and then borg.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200504/8b5e35c4/attachment.html>

From roland at micite.net  Mon May  4 06:59:53 2020
From: roland at micite.net (Roland van Laar)
Date: Mon, 4 May 2020 12:59:53 +0200
Subject: [Borgbackup] Complete download of archive fails - mostly
In-Reply-To: <mailman.5.1588262401.20750.borgbackup@python.org>
References: <mailman.5.1588262401.20750.borgbackup@python.org>
Message-ID: <bbc50691-b8d6-bd0c-98d0-696a9f08cf18@micite.net>

----------------------------------------------------------------------
> Message: 1
> Date: Wed, 29 Apr 2020 20:53:50 +0200
> From: "Gerrit A. Smit :: AT" <gait at atcomputing.nl>
> To: borgbackup at python.org
> Subject: [Borgbackup] Complete download of archive fails - mostly
> Message-ID:
> 	<CAF4+Mbktx=K5vtcgRbBUk1rbtZ27mvBtsztA2JzGabrQBVSDuQ at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
>
>
> I am using rsync.net as a provider of disk-space (very cheap as no
> ZFS-snapshots are needed).
>
> After uploading the first 10 archives it was about time to test things the
> other way around,
> so I did a borg extract ... --dry-run ...
>
> Only to see that at about 10% of the work it stopped working.
> SSH-server closed connection.
> This was from a Google Compute host.

I've had problems with rsync in the past, don't anymore.
This is in my .ssh/config file:

ServerAliveInterval 60

>
> When I did the same thing from home, with a very poor internet connection,
> it took several hours, but eventually all was OK.
>
> I did try using the pv-wrapper-script, which helped only a bit (at most
> 60%).
>
> What could be wrong 'in the cloud'?
>
> rsync.net support just told me they drastically enlarged my quota, but I
> fail to see how that could help: outside of my host, the internet should
> only see compressed/encrypted/deduplicated contents, right?
>
>


From borgbackup at aluaces.fastmail.com  Mon May  4 12:47:51 2020
From: borgbackup at aluaces.fastmail.com (Alberto Luaces)
Date: Mon, 04 May 2020 18:47:51 +0200
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
Message-ID: <f212dda1-5d2c-466e-8146-52471fe495d8@www.fastmail.com>

On Mon, May 4, 2020, at 11:54, Dmitry Astapov wrote:
> If you clone the repo and try to run the test yourself, you will find that it tries to run a binary called "attic", not "borg".
> So the author chose to rename "borg" to "attic" because ... why exactly? 
Initially it was a fork, so they could not be named identically.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200504/da6e5ef6/attachment.html>

From dastapov at gmail.com  Mon May  4 12:51:58 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Mon, 4 May 2020 17:51:58 +0100
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <f212dda1-5d2c-466e-8146-52471fe495d8@www.fastmail.com>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
 <f212dda1-5d2c-466e-8146-52471fe495d8@www.fastmail.com>
Message-ID: <CAFQUnFiMhU+gq0FfikuhpASw6OnN1b5fbojoMM_ph1i6pOmjNA@mail.gmail.com>

Repo claims that it tested  "1.1.0beta6", which was well after the
fork/rename.

On Mon, May 4, 2020 at 5:48 PM Alberto Luaces <
borgbackup at aluaces.fastmail.com> wrote:

> On Mon, May 4, 2020, at 11:54, Dmitry Astapov wrote:
>
> If you clone the repo and try to run the test yourself, you will find that
> it tries to run a binary called "attic", not "borg".
> So the author chose to rename "borg" to "attic" because ... why exactly?
>
> Initially it was a fork, so they could not be named identically.
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>


-- 
D. Astapov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200504/5918598a/attachment.html>

From mrobti at insiberia.net  Mon May  4 13:55:18 2020
From: mrobti at insiberia.net (MRob)
Date: Mon, 04 May 2020 17:55:18 +0000
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
Message-ID: <6d57be6399e234ba51df4b89269a0d0f@insiberia.net>

Dmitry thank you for your perspective,

>> Maybe borg improved from attic, but its not the point. Borg, attic,
>> duplicacy based on deduplication using massive higher storage space in
>> relation to duplicity (and rdiff-backup?). I dont' understand why
>> file-based delta is more storage efficient then deduplication which 
>> can
>> consolidate chunks from all files in the repo. I expect the opposite
>> storage use ratio comparison.
> 
> In real-life scenarios when taking full backups could be prohibitively
> expensive (especially scenarios like massive amounts of data over
> high-latency low-speed links), rdiff-backup/duplicity approach becomes
> simply unviable, disk space saving or not, because you either have to 
> keep
> all backups infinitely long (and eventually run out of storage space) 
> or
> you need to pay the price of full backup once in a while (which will 
> likely
> overshadow all the time/disk space savings you have made previously).

I understand. For my case latency is not a concern. Also most files are 
text (for server backup not personal media). I use rsnapshot so I can 
take backup strategys for 1y, 6m, daily, etc. It is hard-link base so it 
is not so bad for disk space but I am exploring better choices, so far 
Borg looks best.

Yet I still want to understand if it is true or not true that 
deduplication would reduce disk space requirement. Isn't it the purpose 
of deduplication? Even if compression choice was not fairly evaluated in 
that comparison, why doesn't deduplication plus (worse) compression come 
closer to duplicity?

My first test showing with text data the default compression works nice 
(near 40% reduction) but common chunks is not very good (I think "total 
chunks" minus "unique chunks" right?), a very small reduction (1%) from 
deduplication. First time transfer was also slow (even on fast local 
link) but if its a one time operation thats ok.

> So some fraction of extra space taken by bog will be manifests that tie
> blocks in the storage to files that use them.

I don't mind overhead but wnat to have clear understand of the costs and 
benefit. Your opinion is for big picture having variety of backup 
snapshots. I agree to take the big perspective but also can you help me 
understand the details picture, if deduplication does better with 
storage than others.

Compare to hardlink setup like rsnapshot where changed files cause 
entire new file to be kept, I expect because common blocks are kept only 
once the storage reduction would be massive(? is it correct???) But 
similar to be true for rdiff tools that only store deltas. Evaluating 
features of borg vs. rdiff-backup/duplicity borg is clear winner but the 
large difference in storage cost I saw in that comparison is too big to 
ignore.

Another question: after for example a year of backup does it require 
alot more CPU/RAM to compute chunk deltas? What are those costs?

From mrobti at insiberia.net  Mon May  4 14:04:30 2020
From: mrobti at insiberia.net (MRob)
Date: Mon, 04 May 2020 18:04:30 +0000
Subject: [Borgbackup] Offload CPU/RAM cost to target?
Message-ID: <a2b636fab059f2566bf2e53dc0412056@insiberia.net>

I know for like encryption you have to do it on the source side. In case 
where backup is local non-encryption are there technique that can 
offload more CPU/RAM requirements (compression?) to the target backup 
server?

I think one idea to rsync everything to the backup server and make a 
borg backup of that on same machine but it will require a full extra 
duplicate of the backup material.

From dastapov at gmail.com  Mon May  4 15:10:44 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Mon, 4 May 2020 20:10:44 +0100
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <6d57be6399e234ba51df4b89269a0d0f@insiberia.net>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
 <6d57be6399e234ba51df4b89269a0d0f@insiberia.net>
Message-ID: <CAFQUnFi=HQkffFO58S+giw0yE1p6s8zYeLhxn9z+3v6Ef=G4gg@mail.gmail.com>

On Mon, May 4, 2020 at 6:55 PM MRob <mrobti at insiberia.net> wrote:

> Dmitry thank you for your perspective,
>
> >> Maybe borg improved from attic, but its not the point. Borg, attic,
> >> duplicacy based on deduplication using massive higher storage space in
> >> relation to duplicity (and rdiff-backup?). I dont' understand why
> >> file-based delta is more storage efficient then deduplication which
> >> can
> >> consolidate chunks from all files in the repo. I expect the opposite
> >> storage use ratio comparison.
> >
> > In real-life scenarios when taking full backups could be prohibitively
> > expensive (especially scenarios like massive amounts of data over
> > high-latency low-speed links), rdiff-backup/duplicity approach becomes
> > simply unviable, disk space saving or not, because you either have to
> > keep
> > all backups infinitely long (and eventually run out of storage space)
> > or
> > you need to pay the price of full backup once in a while (which will
> > likely
> > overshadow all the time/disk space savings you have made previously).
>
> I understand. For my case latency is not a concern. Also most files are
> text (for server backup not personal media). I use rsnapshot so I can
> take backup strategys for 1y, 6m, daily, etc. It is hard-link base so it
> is not so bad for disk space but I am exploring better choices, so far
> Borg looks best.
>
> Yet I still want to understand if it is true or not true that
> deduplication would reduce disk space requirement. Isn't it the purpose
> of deduplication? Even if compression choice was not fairly evaluated in
> that comparison, why doesn't deduplication plus (worse) compression come
> closer to duplicity?
>

Oh, but it is close. If you use zstd or zlib for compression then (using
files from the benchmark we are discussing) first borg backup will be 179
Mb, which is roughly what duplicity has.


>
> My first test showing with text data the default compression works nice
> (near 40% reduction) but common chunks is not very good (I think "total
> chunks" minus "unique chunks" right?),


if you use "borg info" on the backup, I think it will give you the
information you want (number of unique chunks in this backup vs the number
of shared chunks).


> a very small reduction (1%) from
> deduplication. First time transfer was also slow (even on fast local
> link) but if its a one time operation thats ok.
>
> > So some fraction of extra space taken by bog will be manifests that tie
> > blocks in the storage to files that use them.
>
> I don't mind overhead but wnat to have clear understand of the costs and
> benefit. Your opinion is for big picture having variety of backup
> snapshots. I agree to take the big perspective but also can you help me
> understand the details picture, if deduplication does better with
> storage than others.
>
> Compare to hardlink setup like rsnapshot where changed files cause
> entire new file to be kept, I expect because common blocks are kept only
> once the storage reduction would be massive(? is it correct???)


Yes, this is correct - each unique block is stored at most once.


> But similar to be true for rdiff tools that only store deltas. Evaluating
> features of borg vs. rdiff-backup/duplicity borg is clear winner but the
> large difference in storage cost I saw in that comparison is too big to
> ignore.
>

So one difference which I already covered is that "storing deltas" requires
you to store "base" to which you can apply this delta, and this has
implications for backup removal.

The second difference is that file names/paths are input for computing
deltas, so if you move/rename files, this would generate large deltas,
whereas borg will just see the same blocks all over again and "do nothing".

A third difference is that you can easily find out which portion of backup
is "unique" and which is "shared" -- with deltas, you only know the diff to
the base.

Speaking of diffs, borg could easily diff any two backups, which with
deltas is not straightforward, unless you happen to have just a single base
backup.

If anything happens to your base backup and you detect it, you have no
other choice that to take new base backup (and discard all the deltas you
have to the corrupted base). With borg, if you detect corrupted blocks,
your next backup will just store new copies of the block you lost at a
fraction of the cost.

With hardlinked rsync backups, the information about data blocks shared by
different files is stored in the filesystem datastructures (inodes in case
of ext4, etc). With borg, this information is kept in the data structure
which is written into archive storage for every archive. For small files,
this will require at least ~100B/file + some extra overhead for directories
and such. For Linux kernel tree with ~60K files this would be on the order
of 5-10 Mb (i think), which can easily add up in the context of the
benchmark we are discussing....


>
> Another question: after for example a year of backup does it require
> alot more CPU/RAM to compute chunk deltas?


cpu/time will generally be proportional to the number of chunks you have in
total (it may vary from operation to operation)

Reading
https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html
might
help.


> What are those costs?
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>


-- 
D. Astapov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200504/d4c3fc69/attachment-0001.html>

From lazyvirus at gmx.com  Mon May  4 15:24:55 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Mon, 4 May 2020 21:24:55 +0200
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <CAFQUnFi=HQkffFO58S+giw0yE1p6s8zYeLhxn9z+3v6Ef=G4gg@mail.gmail.com>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
 <6d57be6399e234ba51df4b89269a0d0f@insiberia.net>
 <CAFQUnFi=HQkffFO58S+giw0yE1p6s8zYeLhxn9z+3v6Ef=G4gg@mail.gmail.com>
Message-ID: <20200504212455.6a29460f@msi.defcon1.lan>

On Mon, 4 May 2020 20:10:44 +0100
Dmitry Astapov <dastapov at gmail.com> wrote:

In fact, at this time borg is only missing the possibility of doing
backups from several machines at once in the same repo and, may be,
relocate the clients' index files on the server.

Which raises two questions upon those two points :
* is it possible with the actual borg structure ?
* is it suitable ?

> So one difference which I already covered is that "storing deltas"
> requires you to store "base" to which you can apply this delta, and
> this has implications for backup removal.
>
> The second difference is that file names/paths are input for computing
> deltas, so if you move/rename files, this would generate large deltas,
> whereas borg will just see the same blocks all over again and "do
> nothing".
>
> A third difference is that you can easily find out which portion of
> backup is "unique" and which is "shared" -- with deltas, you only know
> the diff to the base.
>
> Speaking of diffs, borg could easily diff any two backups, which with
> deltas is not straightforward, unless you happen to have just a single
> base backup.
>
> If anything happens to your base backup and you detect it, you have no
> other choice that to take new base backup (and discard all the deltas
> you have to the corrupted base). With borg, if you detect corrupted
> blocks, your next backup will just store new copies of the block you
> lost at a fraction of the cost.
>
> With hardlinked rsync backups, the information about data blocks
> shared by different files is stored in the filesystem datastructures
> (inodes in case of ext4, etc). With borg, this information is kept in
> the data structure which is written into archive storage for every
> archive. For small files, this will require at least ~100B/file + some
> extra overhead for directories and such. For Linux kernel tree with
> ~60K files this would be on the order of 5-10 Mb (i think), which can
> easily add up in the context of the benchmark we are discussing....

From spython01 at gmail.com  Mon May  4 21:18:53 2020
From: spython01 at gmail.com (Samir Parikh)
Date: Mon, 4 May 2020 21:18:53 -0400
Subject: [Borgbackup] Help with Initializing Remote Repository
Message-ID: <CAMj1aXXx+n2=M2hM=14dUur3izLfxOQBpMJiH57XUvmM1ODtyQ@mail.gmail.com>

Hello,

I am trying to back up from a local Ubuntu laptop running borgbackup
version 1.0.12 to a remote Ubuntu VM running borgbackup version
1.1.15.

I created the following environment variable as I am using an
alternate SSH key which does not use a passphrase (to facilitate
future automation):

export BORG_RSH='ssh -i /home/spython01/.ssh/borgsshkey'

I then ran the following command on my local machine to initialize a
remote repo:

borg init --encryption=repokey borg at ip.address:/home/borg/borgbackup

but got the following message, instead of a prompt to create a password:

root Remote: There is already something at /home/borg/borgbackup.
('Remote Exception (see remote log for the traceback)', 'PathAlreadyExists')
Platform: Linux 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10
UTC 2020 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.0.12  Python: CPython 3.5.2
PID: 20952  CWD: /home/spython01
sys.argv: ['/usr/bin/borg', 'init', '--encryption=repokey',
'borg at ip.address:/home/borg/borgbackup']
SSH_ORIGINAL_COMMAND: None

Permissions on the remote VM for the backup directory are as follows:

drwxr-xr-x 3 borg borg 4096 May  4 23:41 borgbackup

Any suggestions on what to change or what I did wrong?

Thanks
Samir

From kmq at omfa.de  Tue May  5 04:09:34 2020
From: kmq at omfa.de (kmq)
Date: Tue, 5 May 2020 10:09:34 +0200
Subject: [Borgbackup] Help with Initializing Remote Repository
In-Reply-To: <CAMj1aXXx+n2=M2hM=14dUur3izLfxOQBpMJiH57XUvmM1ODtyQ@mail.gmail.com>
References: <CAMj1aXXx+n2=M2hM=14dUur3izLfxOQBpMJiH57XUvmM1ODtyQ@mail.gmail.com>
Message-ID: <20200505080934.GA77646@klx>

The problem is likely exactly what the error message is saying:

> root Remote: There is already something at /home/borg/borgbackup.
> ('Remote Exception (see remote log for the traceback)', 'PathAlreadyExists')

In other words:

You already created the directory that borg is trying to initialize,
so borg is helping you by not overwriting everything that might be in
it.

On Mon, May 04, 2020 at 09:18:53PM -0400, Samir Parikh wrote:
> Hello,
>
> I am trying to back up from a local Ubuntu laptop running borgbackup
> version 1.0.12 to a remote Ubuntu VM running borgbackup version
> 1.1.15.
>
> I created the following environment variable as I am using an
> alternate SSH key which does not use a passphrase (to facilitate
> future automation):
>
> export BORG_RSH='ssh -i /home/spython01/.ssh/borgsshkey'
>
> I then ran the following command on my local machine to initialize a
> remote repo:
>
> borg init --encryption=repokey borg at ip.address:/home/borg/borgbackup
>
> but got the following message, instead of a prompt to create a password:
>
> root Remote: There is already something at /home/borg/borgbackup.
> ('Remote Exception (see remote log for the traceback)', 'PathAlreadyExists')
> Platform: Linux 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10
> UTC 2020 x86_64
> Linux: Ubuntu 16.04 xenial
> Borg: 1.0.12  Python: CPython 3.5.2
> PID: 20952  CWD: /home/spython01
> sys.argv: ['/usr/bin/borg', 'init', '--encryption=repokey',
> 'borg at ip.address:/home/borg/borgbackup']
> SSH_ORIGINAL_COMMAND: None
>
> Permissions on the remote VM for the backup directory are as follows:
>
> drwxr-xr-x 3 borg borg 4096 May  4 23:41 borgbackup
>
> Any suggestions on what to change or what I did wrong?
>
> Thanks
> Samir
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup

From spython01 at gmail.com  Tue May  5 09:30:58 2020
From: spython01 at gmail.com (Samir Parikh)
Date: Tue, 5 May 2020 09:30:58 -0400
Subject: [Borgbackup] Help with Initializing Remote Repository
In-Reply-To: <20200505080934.GA77646@klx>
References: <CAMj1aXXx+n2=M2hM=14dUur3izLfxOQBpMJiH57XUvmM1ODtyQ@mail.gmail.com>
 <20200505080934.GA77646@klx>
Message-ID: <CAMj1aXU7W8o+JVJoArzMQFHwjB-dR=rb9+i3YSctGQbZdc2mdg@mail.gmail.com>

On Tue, May 5, 2020 at 4:09 AM kmq <kmq at omfa.de> wrote:
>
> The problem is likely exactly what the error message is saying:
>
> > root Remote: There is already something at /home/borg/borgbackup.
> > ('Remote Exception (see remote log for the traceback)', 'PathAlreadyExists')
>
> In other words:
>
> You already created the directory that borg is trying to initialize,
> so borg is helping you by not overwriting everything that might be in
> it.

I was under the impression that the target directory on the remote
system had to already exist before running the init command.

In the future, should I just let borg create the directory?  (I.e. it
doesn't matter if that directory does not already exist)?

Thank you for your response!

From l0f4r0 at tuta.io  Tue May  5 10:34:00 2020
From: l0f4r0 at tuta.io (l0f4r0 at tuta.io)
Date: Tue, 5 May 2020 16:34:00 +0200 (CEST)
Subject: [Borgbackup] Help with Initializing Remote Repository
In-Reply-To: <CAMj1aXU7W8o+JVJoArzMQFHwjB-dR=rb9+i3YSctGQbZdc2mdg@mail.gmail.com>
References: <CAMj1aXXx+n2=M2hM=14dUur3izLfxOQBpMJiH57XUvmM1ODtyQ@mail.gmail.com>
 <20200505080934.GA77646@klx>
 <CAMj1aXU7W8o+JVJoArzMQFHwjB-dR=rb9+i3YSctGQbZdc2mdg@mail.gmail.com>
Message-ID: <M6_EZA7--3-2@tuta.io>

Hi,

5 mai 2020 ? 15:30 de spython01 at gmail.com:

> On Tue, May 5, 2020 at 4:09 AM kmq <kmq at omfa.de> wrote:
>
>>
>> The problem is likely exactly what the error message is saying:
>>
>> > root Remote: There is already something at /home/borg/borgbackup.
>> > ('Remote Exception (see remote log for the traceback)', 'PathAlreadyExists')
>>
>> In other words:
>>
>> You already created the directory that borg is trying to initialize,
>> so borg is helping you by not overwriting everything that might be in
>> it.
>>
> I was under the impression that the target directory on the remote
> system had to already exist before running the init command.
>
> In the future, should I just let borg create the directory?  (I.e. it
> doesn't matter if that directory does not already exist)?
>
The issue is not about the directory itself, but about its existing content as mentioned by kmq.
You need to create the path first outside of borgbackup, I'm not sure borg can do that for you.

Best regards,
l0f4r0

From spython01 at gmail.com  Tue May  5 11:19:10 2020
From: spython01 at gmail.com (Samir Parikh)
Date: Tue, 5 May 2020 11:19:10 -0400
Subject: [Borgbackup] Help with Initializing Remote Repository
In-Reply-To: <M6_EZA7--3-2@tuta.io>
References: <CAMj1aXXx+n2=M2hM=14dUur3izLfxOQBpMJiH57XUvmM1ODtyQ@mail.gmail.com>
 <20200505080934.GA77646@klx>
 <CAMj1aXU7W8o+JVJoArzMQFHwjB-dR=rb9+i3YSctGQbZdc2mdg@mail.gmail.com>
 <M6_EZA7--3-2@tuta.io>
Message-ID: <CAMj1aXVQnnPLYeoOfYW7etoYsfDb1WkHMt8GA=3XrQ1EH361Xw@mail.gmail.com>

> > I was under the impression that the target directory on the remote
> > system had to already exist before running the init command.
> >
> > In the future, should I just let borg create the directory?  (I.e. it
> > doesn't matter if that directory does not already exist)?
> >
> The issue is not about the directory itself, but about its existing content as mentioned by kmq.
> You need to create the path first outside of borgbackup, I'm not sure borg can do that for you.

Got it.  Thank you!

From mrobti at insiberia.net  Tue May  5 17:24:19 2020
From: mrobti at insiberia.net (MRob)
Date: Tue, 05 May 2020 21:24:19 +0000
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <CAFQUnFi=HQkffFO58S+giw0yE1p6s8zYeLhxn9z+3v6Ef=G4gg@mail.gmail.com>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
 <6d57be6399e234ba51df4b89269a0d0f@insiberia.net>
 <CAFQUnFi=HQkffFO58S+giw0yE1p6s8zYeLhxn9z+3v6Ef=G4gg@mail.gmail.com>
Message-ID: <cf972a2db814dde519b503ee9de4ba30@insiberia.net>

Thanks Dmitry for your time,

>> >> Maybe borg improved from attic, but its not the point. Borg, attic,
>> >> duplicacy based on deduplication using massive higher storage space in
>> >> relation to duplicity (and rdiff-backup?). I dont' understand why
>> >> file-based delta is more storage efficient then deduplication which
>> >> can
>> >> consolidate chunks from all files in the repo. I expect the opposite
>> >> storage use ratio comparison.
>> 
>> Yet I still want to understand if it is true or not true that
>> deduplication would reduce disk space requirement. Isn't it the 
>> purpose
>> of deduplication? Even if compression choice was not fairly evaluated 
>> in
>> that comparison, why doesn't deduplication plus (worse) compression 
>> come
>> closer to duplicity?
> 
> Oh, but it is close. If you use zstd or zlib for compression then 
> (using
> files from the benchmark we are discussing) first borg backup will be 
> 179
> Mb, which is roughly what duplicity has.

Thank you. But this seem to point out that deduplication does not help 
reduce storage size of a set of files alone - that there are so few 
duplicate blocks detected between files. I found in a small test there 
was 1% shared chunks after first backup. Deduplication finds large 
number of shared chunks between archives with the same files, as we 
should expecting (my test, unique chunks does not grow very much so 
shared chunks % always close to 100/number of backups in repo).

I did expected there would be shared chunks between different files too 
as a reason I was expecting deduplication to have a larger storage 
advantage. As I see now, deduplication look similar to other forms of 
delta techniques in term of storage requirement.

Yet I'm surprised and do not understand the part of that study, where 
deduplication software storage growing much faster than rdiff/duplicity 
based software. Duplicacy and Borg grew sometime 1% to 30%+ but 
Duplicity grew never more than 9% sometime less than 1%.

If you think of the question in relativity terms like this can you talk 
about why deduplication storage balloon alot more than rdiff/duplicity 
solution?

You wrote about Borg overhead for many small files like Linux kernel, 
may take ~5-10Mb; is this per every backup archive? Is this part of the 
reason? Is that the cost of the many advantages of Borg?


>> Compare to hardlink setup like rsnapshot where changed files cause
>> entire new file to be kept, I expect because common blocks are kept 
>> only
>> once the storage reduction would be massive(? is it correct???)
> 
> Yes, this is correct - each unique block is stored at most once.

Do you think the conclusion also true, massive storage reduction 
(compare to hardlink technique) because of it?

From mrobti at insiberia.net  Tue May  5 17:33:06 2020
From: mrobti at insiberia.net (MRob)
Date: Tue, 05 May 2020 21:33:06 +0000
Subject: [Borgbackup] CPU/time use of Borg
Message-ID: <31a2335d266fe12ec0366cc5ccd2e436@insiberia.net>

>> Another question: after for example a year of backup does it require
>> alot more CPU/RAM to compute chunk deltas?
> 
> cpu/time will generally be proportional to the number of chunks you 
> have in
> total (it may vary from operation to operation)

My simple test over local network took alittle over 2min/Gb on 1st 
backup. Its ok if only 1st time. But 2nd time was still over 1min/Gb and 
3rd backup very close to 2min/Gb again. (its for busy production server)

I like borg, but if it wont ever get faster maybe I can't use borg :(  
Is borg only for fast & non-busy hardware?

(see other thread, asking way for offload operations from client to 
backup server)

From dastapov at gmail.com  Tue May  5 18:06:05 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Tue, 5 May 2020 23:06:05 +0100
Subject: [Borgbackup] CPU/time use of Borg
In-Reply-To: <31a2335d266fe12ec0366cc5ccd2e436@insiberia.net>
References: <31a2335d266fe12ec0366cc5ccd2e436@insiberia.net>
Message-ID: <CAFQUnFiGCnA3cESMvxmUei44Fy61rDLPJ+FyNt3aAbmZ9dAYQw@mail.gmail.com>

This is how a typical backup looks for me :

Duration: 1 minutes 41.41 seconds
Number of files: 558471
------------------------------------------------------------
------------------
                       Original size      Compressed size    Deduplicated
size
This archive:               78.84 GB             50.79 GB             31.52
MB
All archives:               23.03 TB             14.64 TB            101.51
GB

                       Unique chunks         Total chunks
Chunk index:                  778813            170023300

This is over wifi to a NAS server on the local network.

Backups are taken every 30 minutes, process on the source system uses about
330 Mb VSS / 230 Mb RSS.

There must be something in your setup that causes slowness (source
filesystem? Target filesystem? No stable inodes ? No stable mtimes? ....).

On Tue, May 5, 2020, 22:33 MRob <mrobti at insiberia.net> wrote:

> >> Another question: after for example a year of backup does it require
> >> alot more CPU/RAM to compute chunk deltas?
> >
> > cpu/time will generally be proportional to the number of chunks you
> > have in
> > total (it may vary from operation to operation)
>
> My simple test over local network took alittle over 2min/Gb on 1st
> backup. Its ok if only 1st time. But 2nd time was still over 1min/Gb and
> 3rd backup very close to 2min/Gb again. (its for busy production server)
>
> I like borg, but if it wont ever get faster maybe I can't use borg :(
> Is borg only for fast & non-busy hardware?
>
> (see other thread, asking way for offload operations from client to
> backup server)
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200505/d56f5041/attachment.html>

From dastapov at gmail.com  Tue May  5 18:26:41 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Tue, 5 May 2020 23:26:41 +0100
Subject: [Borgbackup] Storage, CPU, RAM comparisons
In-Reply-To: <cf972a2db814dde519b503ee9de4ba30@insiberia.net>
References: <ab4abd23e2bf27e6bb902a0c75f643bf@insiberia.net>
 <85604ea9-ed4e-4b47-a6c4-baec44d7f028@Spark>
 <CAFQUnFhqCM8ihnPmpm=7t=TB_v1Mu33J1ARL5t8Wx7v-eAnThA@mail.gmail.com>
 <898e273b561646e9808f7d060bcca690@insiberia.net>
 <CAFQUnFg-it4ia8-M2EfqCF6WAVMj4vHeFuvbwVBBekby363vXg@mail.gmail.com>
 <6d57be6399e234ba51df4b89269a0d0f@insiberia.net>
 <CAFQUnFi=HQkffFO58S+giw0yE1p6s8zYeLhxn9z+3v6Ef=G4gg@mail.gmail.com>
 <cf972a2db814dde519b503ee9de4ba30@insiberia.net>
Message-ID: <CAFQUnFi8wHgB-uTWbDMN27H4TpjwGMTB_nf=unj1rKSprRPn7w@mail.gmail.com>

On Tue, May 5, 2020, 22:24 MRob <mrobti at insiberia.net> wrote:

> Thanks Dmitry for your time,
>
> >> >> Maybe borg improved from attic, but its not the point. Borg, attic,
> >> >> duplicacy based on deduplication using massive higher storage space
> in
> >> >> relation to duplicity (and rdiff-backup?). I dont' understand why
> >> >> file-based delta is more storage efficient then deduplication which
> >> >> can
> >> >> consolidate chunks from all files in the repo. I expect the opposite
> >> >> storage use ratio comparison.
> >>
> >> Yet I still want to understand if it is true or not true that
> >> deduplication would reduce disk space requirement. Isn't it the
> >> purpose
> >> of deduplication? Even if compression choice was not fairly evaluated
> >> in
> >> that comparison, why doesn't deduplication plus (worse) compression
> >> come
> >> closer to duplicity?
> >
> > Oh, but it is close. If you use zstd or zlib for compression then
> > (using
> > files from the benchmark we are discussing) first borg backup will be
> > 179
> > Mb, which is roughly what duplicity has.
>
> Thank you. But this seem to point out that deduplication does not help
> reduce storage size of a set of files alone - that there are so few
> duplicate blocks detected between files. I found in a small test there
> was 1% shared chunks after first backup. Deduplication finds large
> number of shared chunks between archives with the same files, as we
> should expecting (my test, unique chunks does not grow very much so
> shared chunks % always close to 100/number of backups in repo).
>
> I did expected there would be shared chunks between different files too
> as a reason I was expecting deduplication to have a larger storage
> advantage. As I see now, deduplication look similar to other forms of
> delta techniques in term of storage requirement.
>

That of course depends on your files. I found that for VM images/snapshots,
multiple copies of the photos with different retouching applied, large text
files (logs), etc there is a good deal of sharing



> Yet I'm surprised and do not understand the part of that study, where
> deduplication software storage growing much faster than rdiff/duplicity
> based software. Duplicacy and Borg grew sometime 1% to 30%+ but
> Duplicity grew never more than 9% sometime less than 1%.
>

Assuming we are still talking about that github project, you are looking at
the archive of less than 1gb in size, and only because of this small data
size it is possible for the metadata for 12 backups to occupy a sizeable
portion of total size.

Thinking in percents is useless, this will not hold for large archives.


> If you think of the question in relativity terms like this can you talk
> about why deduplication storage balloon alot more than rdiff/duplicity
> solution?


Well, it does not, not for me.


> You wrote about Borg overhead for many small files like Linux kernel,
> may take ~5-10Mb; is this per every backup archive?


Yes, it would be

Is this part of the
> reason?


Yes, I suppose.

Is that the cost of the many advantages of Borg?
>

Is this really a cost? I suppose it could be viewed as a cost only if you
backup megabytes, not hundreds of gigabytes.



>
> >> Compare to hardlink setup like rsnapshot where changed files cause
> >> entire new file to be kept, I expect because common blocks are kept
> >> only
> >> once the storage reduction would be massive(? is it correct???)
> >
> > Yes, this is correct - each unique block is stored at most once.
>
> Do you think the conclusion also true, massive storage reduction
> (compare to hardlink technique) because of it?
>


Sorry, I failed to parse this.

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200505/0720d946/attachment-0001.html>

From gait at atcomputing.nl  Wed May  6 04:08:35 2020
From: gait at atcomputing.nl (Gerrit A. Smit :: AT)
Date: Wed, 6 May 2020 10:08:35 +0200
Subject: [Borgbackup] Complete download of archive fails - mostly
In-Reply-To: <bbc50691-b8d6-bd0c-98d0-696a9f08cf18@micite.net>
References: <mailman.5.1588262401.20750.borgbackup@python.org>
 <bbc50691-b8d6-bd0c-98d0-696a9f08cf18@micite.net>
Message-ID: <CAF4+Mbk=im7CP9-Jx_uxAnQToAZ-Ujd8pq3HcPFUzY6dn67fUg@mail.gmail.com>

Hello,

In the meantime I learned that the Borg server on my side,
which is a Google Compute Instance,
was 'OOM killed' a few times, so that will probably be the cause of all
this trouble.

This is in my SSH-config:

Host rsync_net
       IdentityFile    ...
       Hostname        ...
       User            ...
       BatchMode       yes
       ServerAliveInterval 30
       ServerAliveCountMax 3
       UseRoaming      no
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200506/e5ed2af9/attachment.html>

From mrobti at insiberia.net  Wed May  6 18:33:24 2020
From: mrobti at insiberia.net (MRob)
Date: Wed, 06 May 2020 22:33:24 +0000
Subject: [Borgbackup] CPU/time use of Borg
In-Reply-To: <CAFQUnFiGCnA3cESMvxmUei44Fy61rDLPJ+FyNt3aAbmZ9dAYQw@mail.gmail.com>
References: <31a2335d266fe12ec0366cc5ccd2e436@insiberia.net>
 <CAFQUnFiGCnA3cESMvxmUei44Fy61rDLPJ+FyNt3aAbmZ9dAYQw@mail.gmail.com>
Message-ID: <e19c06caf9324105db79f254d5dd69c9@insiberia.net>

On 2020-05-05 22:06, Dmitry Astapov wrote:
> This is how a typical backup looks for me :

Thank you Dmitry this is helpful! My testing on very busy server 
probably starved CPU and some iowait. Your stats helpful to see. Am i 
right it is personal comp (not very busy CPU)?

> Duration: 1 minutes 41.41 seconds
> Number of files: 558471
> ------------------------------------------------------------
> ------------------
>                        Original size      Compressed size    
> Deduplicated
> size
> This archive:               78.84 GB             50.79 GB             
> 31.52
> MB
> All archives:               23.03 TB             14.64 TB            
> 101.51
> GB
> 
>                        Unique chunks         Total chunks
> Chunk index:                  778813            170023300
> 
> This is over wifi to a NAS server on the local network.
> 
> Backups are taken every 30 minutes, process on the source system uses 
> about
> 330 Mb VSS / 230 Mb RSS.
> 
> There must be something in your setup that causes slowness (source
> filesystem? Target filesystem? No stable inodes ? No stable mtimes? 
> ....).
> 
> On Tue, May 5, 2020, 22:33 MRob <mrobti at insiberia.net> wrote:
> 
>> >> Another question: after for example a year of backup does it require
>> >> alot more CPU/RAM to compute chunk deltas?
>> >
>> > cpu/time will generally be proportional to the number of chunks you
>> > have in
>> > total (it may vary from operation to operation)
>> 
>> My simple test over local network took alittle over 2min/Gb on 1st
>> backup. Its ok if only 1st time. But 2nd time was still over 1min/Gb 
>> and
>> 3rd backup very close to 2min/Gb again. (its for busy production 
>> server)
>> 
>> I like borg, but if it wont ever get faster maybe I can't use borg :(
>> Is borg only for fast & non-busy hardware?
>> 
>> (see other thread, asking way for offload operations from client to
>> backup server)
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
>> 

From alessandro.baggi at gmail.com  Thu May  7 03:18:04 2020
From: alessandro.baggi at gmail.com (Alessandro Baggi)
Date: Thu, 7 May 2020 09:18:04 +0200
Subject: [Borgbackup] borgbackup version between distro
Message-ID: <54bfca09-a939-5072-b43c-7b08e3fe647d@gmail.com>

Hi list,
I'm new to borgbackup and I'm planning to try at home for my little lan.

I have a backup server running centos 8, a nas running debian 10 and my 
workstation where I would run arch.

Currently I have:

- Centos 8 with borg 1.1.11 (my central repository)
- Debian 10 with borg 1.1.9 (client)
- Arch with borg 1.1.11 (if I'm not wrong) (client)

Because I have not any knowledge about using multiple version of borg 
with remote repository I could have problem about using multiple version 
of borg with a central repository?

What are the best practices of using multiple borgbackup version together?

I ask this because, in the past I had some problem using bacula with 
different version (5.4, 7.0 and 9.2) and caused me much problem.

Thank you in advance.

From clickwir at gmail.com  Thu May  7 10:55:31 2020
From: clickwir at gmail.com (Zack Coffey)
Date: Thu, 7 May 2020 08:55:31 -0600
Subject: [Borgbackup] borgbackup version between distro
In-Reply-To: <54bfca09-a939-5072-b43c-7b08e3fe647d@gmail.com>
References: <54bfca09-a939-5072-b43c-7b08e3fe647d@gmail.com>
Message-ID: <CAA8KC9JnegmNQHXJPi7m76zSrD+gYpQpuDLci87VpKsdhgZSag@mail.gmail.com>

Short answer, you'll be fine. I've used different versions in the 1.1.x
line and everything has worked fine.


Longer answer; I'm not surprised you had trouble with other software with
such different versions being incompatible. Generally speaking, for the
most part, software development and version numbers follow some common
sense. Meaning, from left to right you have the MAJOR.MINOR.PATCH versions.

Some software will either be completely or significantly incompatible
between major versions. Less often, but can happen, you'll find some sort
of incompatibility in minor versions. But for the most part, when talking
about differences in patch versions they are compatible. Note, I've said
'generally' and 'most part' and 'some' because not all software follows a
strict rule.

Now as for best practices, have all your versions match. There are
situations that could prevent you from running matched versions, but
getting them all to be the same or as close as possible is ideal.

You can read more here about the general idea of version number methods.
https://en.wikipedia.org/wiki/Software_versioning


On Thu, May 7, 2020 at 1:18 AM Alessandro Baggi <alessandro.baggi at gmail.com>
wrote:

> Hi list,
> I'm new to borgbackup and I'm planning to try at home for my little lan.
>
> I have a backup server running centos 8, a nas running debian 10 and my
> workstation where I would run arch.
>
> Currently I have:
>
> - Centos 8 with borg 1.1.11 (my central repository)
> - Debian 10 with borg 1.1.9 (client)
> - Arch with borg 1.1.11 (if I'm not wrong) (client)
>
> Because I have not any knowledge about using multiple version of borg
> with remote repository I could have problem about using multiple version
> of borg with a central repository?
>
> What are the best practices of using multiple borgbackup version together?
>
> I ask this because, in the past I had some problem using bacula with
> different version (5.4, 7.0 and 9.2) and caused me much problem.
>
> Thank you in advance.
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200507/ddff1184/attachment.html>

From alessandro.baggi at gmail.com  Thu May  7 11:27:30 2020
From: alessandro.baggi at gmail.com (Alessandro Baggi)
Date: Thu, 7 May 2020 17:27:30 +0200
Subject: [Borgbackup] borgbackup version between distro
In-Reply-To: <CAA8KC9JnegmNQHXJPi7m76zSrD+gYpQpuDLci87VpKsdhgZSag@mail.gmail.com>
References: <54bfca09-a939-5072-b43c-7b08e3fe647d@gmail.com>
 <CAA8KC9JnegmNQHXJPi7m76zSrD+gYpQpuDLci87VpKsdhgZSag@mail.gmail.com>
Message-ID: <01bf5feb-e17f-66db-0e83-f8d8f6f7e36d@gmail.com>

Thank you for your suggestions.

Il 07/05/20 16:55, Zack Coffey ha scritto:
> Short answer, you'll be fine. I've used different versions in the 1.1.x 
> line and everything has worked fine.
> 
> 
> Longer answer; I'm not surprised you had trouble with other software 
> with such different versions being incompatible. Generally?speaking, for 
> the most part, software development and version numbers follow some 
> common sense. Meaning, from left to?right you have the MAJOR.MINOR.PATCH 
> versions.
> 
> Some software will either be completely or significantly incompatible 
> between major versions. Less often, but can happen, you'll find some 
> sort of incompatibility?in minor versions. But for the most part, when 
> talking about differences in patch versions they are compatible. Note, 
> I've said 'generally' and 'most part' and 'some' because not all 
> software follows a strict rule.
> 
> Now as for best practices, have all your versions match. There are 
> situations that could prevent you from running matched versions, but 
> getting them all to be the same or as close as possible is ideal.
> 
> You can read more here about the general idea of version number methods. 
> https://en.wikipedia.org/wiki/Software_versioning
> 
> 
> On Thu, May 7, 2020 at 1:18 AM Alessandro Baggi 
> <alessandro.baggi at gmail.com <mailto:alessandro.baggi at gmail.com>> wrote:
> 
>     Hi list,
>     I'm new to borgbackup and I'm planning to try at home for my little lan.
> 
>     I have a backup server running centos 8, a nas running debian 10 and my
>     workstation where I would run arch.
> 
>     Currently I have:
> 
>     - Centos 8 with borg 1.1.11 (my central repository)
>     - Debian 10 with borg 1.1.9 (client)
>     - Arch with borg 1.1.11 (if I'm not wrong) (client)
> 
>     Because I have not any knowledge about using multiple version of borg
>     with remote repository I could have problem about using multiple
>     version
>     of borg with a central repository?
> 
>     What are the best practices of using multiple borgbackup version
>     together?
> 
>     I ask this because, in the past I had some problem using bacula with
>     different version (5.4, 7.0 and 9.2) and caused me much problem.
> 
>     Thank you in advance.
>     _______________________________________________
>     Borgbackup mailing list
>     Borgbackup at python.org <mailto:Borgbackup at python.org>
>     https://mail.python.org/mailman/listinfo/borgbackup
> 

From dastapov at gmail.com  Thu May  7 16:26:40 2020
From: dastapov at gmail.com (Dmitry Astapov)
Date: Thu, 7 May 2020 21:26:40 +0100
Subject: [Borgbackup] CPU/time use of Borg
In-Reply-To: <e19c06caf9324105db79f254d5dd69c9@insiberia.net>
References: <31a2335d266fe12ec0366cc5ccd2e436@insiberia.net>
 <CAFQUnFiGCnA3cESMvxmUei44Fy61rDLPJ+FyNt3aAbmZ9dAYQw@mail.gmail.com>
 <e19c06caf9324105db79f254d5dd69c9@insiberia.net>
Message-ID: <CAFQUnFh8-dk+ZafuY5ix5w_5wm-D092qSyB8gQ15jDqwsB9_hQ@mail.gmail.com>

On Wed, May 6, 2020 at 11:33 PM MRob <mrobti at insiberia.net> wrote:

> On 2020-05-05 22:06, Dmitry Astapov wrote:
> > This is how a typical backup looks for me :
>
> Thank you Dmitry this is helpful! My testing on very busy server
> probably starved CPU and some iowait. Your stats helpful to see. Am i
> right it is personal comp (not very busy CPU)?
>

Yes, that was from a personal notebook.

Here is a different data set on a moderately loaded server (LA 2-3 most of
the time), backup is done onto the local filesystem:

Archive name: 2020-05-07_02.03
Archive fingerprint:
ac751598e6b99140e64dcf4527993ccb64d6588cc0195a5940e2c7595ccfa9ec
Time (start): Thu, 2020-05-07 02:03:12
Time (end):   Thu, 2020-05-07 02:12:14
Duration: *9 minutes 1.69 seconds*
Number of files: 163791
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated
size
This archive:                *1.48 TB*              1.48 TB
 6.73 MB
All archives:               66.21 TB             66.21 TB              1.36
TB

                       Unique chunks         Total chunks
Chunk index:                  655243             31642036
------------------------------------------------------------------------------

-- 
D. Astapov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200507/b91dd326/attachment.html>

From mrobti at insiberia.net  Thu May  7 17:33:33 2020
From: mrobti at insiberia.net (MRob)
Date: Thu, 07 May 2020 21:33:33 +0000
Subject: [Borgbackup] CPU/time use of Borg
In-Reply-To: <CAFQUnFh8-dk+ZafuY5ix5w_5wm-D092qSyB8gQ15jDqwsB9_hQ@mail.gmail.com>
References: <31a2335d266fe12ec0366cc5ccd2e436@insiberia.net>
 <CAFQUnFiGCnA3cESMvxmUei44Fy61rDLPJ+FyNt3aAbmZ9dAYQw@mail.gmail.com>
 <e19c06caf9324105db79f254d5dd69c9@insiberia.net>
 <CAFQUnFh8-dk+ZafuY5ix5w_5wm-D092qSyB8gQ15jDqwsB9_hQ@mail.gmail.com>
Message-ID: <4e3206f3a0f1a3638913273df7efe2b2@insiberia.net>


On 2020-05-07 20:26, Dmitry Astapov wrote:
> On Wed, May 6, 2020 at 11:33 PM MRob <mrobti at insiberia.net> wrote:
> 
>> On 2020-05-05 22:06, Dmitry Astapov wrote:
>> > This is how a typical backup looks for me :
>> 
>> Thank you Dmitry this is helpful! My testing on very busy server
>> probably starved CPU and some iowait. Your stats helpful to see. Am i
>> right it is personal comp (not very busy CPU)?
>> 
> 
> Yes, that was from a personal notebook.
> 
> Here is a different data set on a moderately loaded server (LA 2-3 most 
> of
> the time), backup is done onto the local filesystem:

Thank you its very generous. Also impressive stats!



> Archive name: 2020-05-07_02.03
> Archive fingerprint:
> ac751598e6b99140e64dcf4527993ccb64d6588cc0195a5940e2c7595ccfa9ec
> Time (start): Thu, 2020-05-07 02:03:12
> Time (end):   Thu, 2020-05-07 02:12:14
> Duration: *9 minutes 1.69 seconds*
> Number of files: 163791
> Utilization of max. archive size: 0%
> ------------------------------------------------------------------------------
>                        Original size      Compressed size    
> Deduplicated
> size
> This archive:                *1.48 TB*              1.48 TB
>  6.73 MB
> All archives:               66.21 TB             66.21 TB              
> 1.36
> TB
> 
>                        Unique chunks         Total chunks
> Chunk index:                  655243             31642036
> ------------------------------------------------------------------------------

From MatthiasPeterW at aol.com  Fri May  8 18:20:17 2020
From: MatthiasPeterW at aol.com (Matthias Peter Walther)
Date: Sat, 9 May 2020 00:20:17 +0200
Subject: [Borgbackup] Borg + tardigrade.io
References: <499c1d42-50fb-eecf-3dfb-e57440eddbf2.ref@aol.com>
Message-ID: <499c1d42-50fb-eecf-3dfb-e57440eddbf2@aol.com>

Hello fellow Borg-users,

I wonder if it would make sense to use borg backup with tardigrade.io.

Technically, it would work, I guess, as i can be S3-like mounted. But
their business modell charges for downloading data.

So I wonder how much data the Borg client would have to read back during
operations, as they charge 4,5 Cents for each GB read from their storage.

Let's say I have 100 Gb which change rarely, mostly just data appended.

- initial backup: probably 100 GB upload plus some overhead, round about
1 $/month so far

?- append: Does the borg client store the state and the hashes to the
chunks or would all data have to be read back?

- prune: Same question here. What is stored on the client side, which
data would have to be read back?

Best,
Matthias


From benibilme at gmail.com  Fri May 15 23:09:03 2020
From: benibilme at gmail.com (Bilinmek Istemiyor)
Date: Sat, 16 May 2020 06:09:03 +0300
Subject: [Borgbackup] remote config and cache directories
Message-ID: <CAB=HgN300MfR-b6vLhGOWPoXLT1tCS0aMmHZFqSTmn7WcfyJGw@mail.gmail.com>

Hello,

I am using borg 1.1.11 on Manjaro system.

I used to use  mounted remote nfs drives to backup my home directory to my
home server. It was very very slow, and I  have decided to switch to ssh.
Passwordless login has been setup with my labtop and home server.

I have following environment variables setup:

export BORG_BASE_DIR=username at myserver.com:
/pool/backups/users/username/borg/home
export BORG_CACHE_DIR=${BORG_BASE_DIR}/cache
export BORG_CONFIG_DIR=${BORG_BASE_DIR}/config
export BORG_REPO=${BORG_BASE_DIR}/repo

However, everytime I run borg create command, borg creates a deep directory
tree named username at myserver.com following the path as described in the
BORG_BASE_DIR path as follows.

username at myserver.com:
    ??? pool        ??? backups
            ??? users                ??? username
                    ??? borg
                        ??? home
                            ??? cache
                            ?   ??? CACHEDIR.TAG
                            ??? config
??? security
                                    ???
5e0cc1a1aa56bf8d4345721927f63c6ffcc1176054ae69d82bc4e459ded90c24
                                        ??? key-type
                                        ??? location
                                        ??? manifest-timestamp


I using borg  with borgmatic generally, and I posted this question to
borgmatic issue tracker, I was informed that it is because of borg, borg
expects config and cache directories to be local.

I have verified this behaviour with borg.

Security is no concern for me,  my repositories are unencrypted, I would
like keep config and cache dirs alongside with the repo. I do not want to
keep borg configuration and cache in my laptop where remote repository is
on remote server.

Isn't it possible to use remote cache and config directories?

Any comment much appreciated.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200516/4aebb319/attachment.html>

From czvhcflh1ruo5q61 at sh00.de  Sat May 16 05:37:52 2020
From: czvhcflh1ruo5q61 at sh00.de (Simon Hoffmann)
Date: Sat, 16 May 2020 11:37:52 +0200
Subject: [Borgbackup] Borg filling local disk completely during backup
Message-ID: <20200516093752.GA24534@SIMON-01>



Hey guys,
?
apologies if this has been asked before, I'm new to borg and the mailinglist.
?
I've set up borg on my mailserver to run every 10 minutes and to keep a rather large amount of archives before pruning. This worked fine for the last weeks, however, a few days ago I found my mailserver's harddrive to be 100% used and thus no new mails arriving.
?
Looking at the drive, I found ~50 GB files in /root/.cache/borg/xxx/chunks.archive.d. (yes, borg runs as root)
?
Since the first goal is to ensure operation, I just deleted the folder.
?
When the borg script ran again a few minutes later, I saw that this folder was again filling up quite fast, so I killed the borg script and disabled it for now.
?
I did not yet run a repo check, I will do that now.
?
Do you know what the problem might be?
?
Here is what I got when running the borg script for the last time:
?
?
?
Synchronizing chunks cache...
Archives: 2049, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 2049.
Fetching and building archive index for mx01-2020-05-06-103002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-005002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-070002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-070001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-165001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-224001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-161001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-015001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-27-192002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-173001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-051001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-222001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-044001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-125001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-032001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-090002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-113001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-150001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-042001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-094001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-035001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-061001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-023001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-223001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-153002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-125001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-113001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-093001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-115002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-023001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-132002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-165001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-03-175002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-03-220001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-232001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-231002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-03-201001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-004001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-000001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-103002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-112001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-132001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-090002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-040001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-225001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-203001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-114002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-170002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-093001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-101001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-231001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-025001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-051001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-143002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-27-151001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-181001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-200001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-092002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-003001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-043001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-010001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-025001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-193001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-151001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-194001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-081001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-204001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-174002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-190001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-074001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-022001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-213001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-134001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-074002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-021001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-152001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-144001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-002001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-03-081001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-013001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-185001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-161001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-214001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-033002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-110001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-175001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-015001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-031002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-032001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-201002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-070001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-182002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-084002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-121001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-184001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-055001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-135001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-134002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-152001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-054001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-105002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-051001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-065001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-221001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-100001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-041001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-132002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-27-163001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-221001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-083001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-191001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-192002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-135002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-27-191001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-192001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-110001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-03-225001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-184001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-170001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-023001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-01-235002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-034001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-052002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-171001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-063002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-084002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-073002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-011001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-171001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-09-214001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-083001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-082002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-012001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-125002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-142001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-004001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-111002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-223001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-192001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-103002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-162001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-003001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-233001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-19-235002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-231001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-174001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-215001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-211001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-28-223002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-082001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-111001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-033001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-011001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-061001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-235001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-140001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-08-094001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-185001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-03-003001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-013001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-223001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-233001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-050001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-03-31-235001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-174001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-084001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-06-040002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-01-170002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-02-175001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-011002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-201001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-020001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-052001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-032001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-27-233001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-11-033002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-023002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-04-001002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-232001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-205002 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-07-053001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-05-202001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-30-172001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-29-163001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-05-10-083001 ...
Merging into master chunks index ...
Fetching and building archive index for mx01-2020-04-27-173002 ...
?
?
Thanks!
?
cheers,
Simon

From kramski at hoernle-marbach.de  Sat May 16 07:55:25 2020
From: kramski at hoernle-marbach.de (Heinz Werner Kramski-Grote)
Date: Sat, 16 May 2020 13:55:25 +0200
Subject: [Borgbackup] Borg filling local disk completely during backup
In-Reply-To: <20200516093752.GA24534@SIMON-01>
References: <20200516093752.GA24534@SIMON-01>
Message-ID: <6944630.6B1Pxy8hQg@sandl>

On Samstag, 16. Mai 2020 11:37:52 CEST Simon Hoffmann via Borgbackup wrote:
> 
> Hey guys,
>  
> apologies if this has been asked before, I'm new to borg and the mailinglist.
>  
> I've set up borg on my mailserver to run every 10 minutes and to keep a rather large amount of archives before pruning. This worked fine for the last weeks, however, a few days ago I found my mailserver's harddrive to be 100% used and thus no new mails arriving.
>  
> Looking at the drive, I found ~50 GB files in /root/.cache/borg/xxx/chunks.archive.d. (yes, borg runs as root)

Are you using dhcpdc? For some reason, dhcpdc 9 started mirroring /proc, /sys etc. below /var/lib/dhcpcd, which leads to read errors and probably huge backups in Borg (https://www.reddit.com/r/archlinux/comments/gaefzl/watch_out_for_backup_issues_with_dhcpcd_90/).

You probably want to exclude
        - /var/lib/dhcpcd/dev
        - /var/lib/dhcpcd/proc
        - /var/lib/dhcpcd/run
        - /var/lib/dhcpcd/sys
by default now.

Regards,
   Heinz



From czvhcflh1ruo5q61 at sh00.de  Sat May 16 09:36:54 2020
From: czvhcflh1ruo5q61 at sh00.de (Simon Hoffmann)
Date: Sat, 16 May 2020 15:36:54 +0200
Subject: [Borgbackup] Borg filling local disk completely during backup
In-Reply-To: <6944630.6B1Pxy8hQg@sandl>
References: <20200516093752.GA24534@SIMON-01>
 <6944630.6B1Pxy8hQg@sandl>
Message-ID: <20200516133654.GA27951@SIMON-01>

> Are you using dhcpdc? For some reason, dhcpdc 9 started mirroring /proc, /sys etc. below /var/lib/dhcpcd, which leads to read errors and probably huge backups in Borg (https://www.reddit.com/r/archlinux/comments/gaefzl/watch_out_for_backup_issues_with_dhcpcd_90/).
> 
> You probably want to exclude
>         - /var/lib/dhcpcd/dev
>         - /var/lib/dhcpcd/proc
>         - /var/lib/dhcpcd/run
>         - /var/lib/dhcpcd/sys
> by default now.

I'm sorry, I forgot to mention system details. The system is a Debian 9 VM and I only
include /etc and /home in the backup, excluding caches. I have configured the system to use a static
address, but I will check for dhcpdc later.

thanks!

cheers,
Simon



From tw at waldmann-edv.de  Sat May 16 09:40:47 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 16 May 2020 15:40:47 +0200
Subject: [Borgbackup] remote config and cache directories
In-Reply-To: <CAB=HgN300MfR-b6vLhGOWPoXLT1tCS0aMmHZFqSTmn7WcfyJGw@mail.gmail.com>
References: <CAB=HgN300MfR-b6vLhGOWPoXLT1tCS0aMmHZFqSTmn7WcfyJGw@mail.gmail.com>
Message-ID: <070c497e-2ccb-83ff-1f77-abce86fd6483@waldmann-edv.de>

> export
> BORG_BASE_DIR=username at myserver.com:/pool/backups/users/username/borg/home

It's called "_DIR" because it wants a (local) directory there.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Sat May 16 09:39:18 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 16 May 2020 15:39:18 +0200
Subject: [Borgbackup] Borg filling local disk completely during backup
In-Reply-To: <20200516093752.GA24534@SIMON-01>
References: <20200516093752.GA24534@SIMON-01>
Message-ID: <bb874184-fd5a-c478-b0dd-32b87d3ebed0@waldmann-edv.de>

> Looking at the drive, I found ~50 GB files in /root/.cache/borg/xxx/chunks.archive.d. (yes, borg runs as root)

This is documented, search the docs for chunks.archive.dir and the misc.
BORG_*_DIR settings.

You can:
- have the cache at the default location (as now)
- have the cache at another location (BORG_CACHE_DIR)
- not have the cache (create file of same name as chunks.archive.d -
will be slower.

Also, you can "borg prune" to have less backups. The more backups you
keep, the bigger that directory might get.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From czvhcflh1ruo5q61 at sh00.de  Sat May 16 11:13:14 2020
From: czvhcflh1ruo5q61 at sh00.de (Simon Hoffmann)
Date: Sat, 16 May 2020 17:13:14 +0200
Subject: [Borgbackup] Borg filling local disk completely during backup
In-Reply-To: <bb874184-fd5a-c478-b0dd-32b87d3ebed0@waldmann-edv.de>
References: <20200516093752.GA24534@SIMON-01>
 <bb874184-fd5a-c478-b0dd-32b87d3ebed0@waldmann-edv.de>
Message-ID: <20200516151313.GA29129@SIMON-01>

> > Looking at the drive, I found ~50 GB files in /root/.cache/borg/xxx/chunks.archive.d. (yes, borg runs as root)
> 
> This is documented, search the docs for chunks.archive.dir and the misc.
> BORG_*_DIR settings.

Yeah, but I'm still not really sure why this happened. For weeks, the disk usage did
not change at all (and thus chunks.archive.d dig not get any bigger), and all of a
sudden borg uses that directory and fills up the entire disk within minutes.

So I'm not looking at a slight increase of disk usage over time (I monitored the disk
usage), but of a sudden increase where the chunks.archive.d gets as big as the data
that needs to be backed-up.

The only thing I could find about chunks.archive.d in the manual was in
internals/data-structure. So the reason for this behaviour is that all of a sudden
the whole repo became corrupted and borg had to rebuild all indices?


 
> Also, you can "borg prune" to have less backups. The more backups you
> keep, the bigger that directory might get.

As I said, I deliberately had a high threshold for pruning, but disk usage never
increased before. I have, however, now revised my pruning settings and will keep less
archives in the future.


I have now created a second repo to backup the server again, but I would still like
to know why this happened.

Thanks!

Cheers, 

Simon



From spython01 at gmail.com  Sun May 17 10:51:35 2020
From: spython01 at gmail.com (Samir Parikh)
Date: Sun, 17 May 2020 10:51:35 -0400
Subject: [Borgbackup] Getting Started Tutorial
Message-ID: <b436fe7a-d00e-b8d4-7781-86b5948868a6@gmail.com>

I know that this project already has excellent documentation, including 
a very helpful "Quick Start" section, and that many people have written 
about their experiences with Borg before, but I thought I would share a 
write up I just did documenting how I set up Borg for myself:

https://samirparikh.com/blog/borgbackup.html

I'm thankful to all of you who provided answers to my questions here 
previously and am hoping I can contribute back in some small way if 
anyone finds this helpful.  Of course, if you find any errors or have 
any suggestions, please fire away!

Thanks again to all of the developers for providing this great tool.

Thanks
Samir

From avkaplmkt at gmail.com  Sun May 17 13:09:04 2020
From: avkaplmkt at gmail.com (Advrk Aplmrkt)
Date: Sun, 17 May 2020 18:09:04 +0100
Subject: [Borgbackup] Getting Started Tutorial
In-Reply-To: <b436fe7a-d00e-b8d4-7781-86b5948868a6@gmail.com>
References: <b436fe7a-d00e-b8d4-7781-86b5948868a6@gmail.com>
Message-ID: <CAN+QS0+YfE67hSgzxvNitF+y8BOv1xhqnp+d7A3sjKXHSWpQ8g@mail.gmail.com>

Thank you, this is a nice summary and a great contribution!

On 17/05/2020, Samir Parikh <spython01 at gmail.com> wrote:
> I know that this project already has excellent documentation, including
> a very helpful "Quick Start" section, and that many people have written
> about their experiences with Borg before, but I thought I would share a
> write up I just did documenting how I set up Borg for myself:
>
> https://samirparikh.com/blog/borgbackup.html
>
> I'm thankful to all of you who provided answers to my questions here
> previously and am hoping I can contribute back in some small way if
> anyone finds this helpful.  Of course, if you find any errors or have
> any suggestions, please fire away!
>
> Thanks again to all of the developers for providing this great tool.
>
> Thanks
> Samir
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>

From avkaplmkt at gmail.com  Sun May 17 13:17:51 2020
From: avkaplmkt at gmail.com (Advrk Aplmrkt)
Date: Sun, 17 May 2020 18:17:51 +0100
Subject: [Borgbackup] Mounted archives only have "r-x" permission for a
 given uid, how to change this?
Message-ID: <CAN+QS0KTbZH=2crR5wF-KHsSZZ8aboqL6qNAfMAZWfVsiQMpNw@mail.gmail.com>

Hello,

I am using the `borg mount` command to mount a borg archive on my
local filesystem. I noticed that the contents of the mounted archive
have the "r-x" permission (read and execute) for the original uid that
they were backed up from.

However, I am now mounting this archive on a separate system under a
user with a different uid, and I can't see or access anything inside
the mounted archive.

When I created the archive, I used the `--umask 0000` option thinking
this will preserve the permissions for other users to see inside it,
but looks like that didn't help. I also tried `--umask 0000` when
mounting the archive but still the mounted contents don't allow any
other uid besides the original one to see inside it.

So, was it something I did wrong when creating the archive or is this
an issue when mounting it? Thanks!

From tw at waldmann-edv.de  Sun May 17 16:37:04 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 17 May 2020 22:37:04 +0200
Subject: [Borgbackup] Borg filling local disk completely during backup
In-Reply-To: <20200516151313.GA29129@SIMON-01>
References: <20200516093752.GA24534@SIMON-01>
 <bb874184-fd5a-c478-b0dd-32b87d3ebed0@waldmann-edv.de>
 <20200516151313.GA29129@SIMON-01>
Message-ID: <0357e554-5bea-1847-89b8-0ac09881b0a7@waldmann-edv.de>



On 2020-05-16 17:13, Simon Hoffmann via Borgbackup wrote:
>>> Looking at the drive, I found ~50 GB files in /root/.cache/borg/xxx/chunks.archive.d. (yes, borg runs as root)
>>
>> This is documented, search the docs for chunks.archive.dir and the misc.
>> BORG_*_DIR settings.
> 
> Yeah, but I'm still not really sure why this happened. For weeks, the disk usage did
> not change at all (and thus chunks.archive.d dig not get any bigger), and all of a
> sudden borg uses that directory and fills up the entire disk within minutes.

If borg detects that the repo is newer than its local chunks index, it
must rebuild this index.

To do that, it downloads metadata of all archives and builds per-archive
indexes and caches them to chunks.archive.d/ (so that, if that happens
again, it only needs to download the new archives' metadata).

>From all these smaller indexes, it builds its master chunks index.

One common cause that triggers this is accessing the repo from some
other client, some other user or locally on the repo server.

> The only thing I could find about chunks.archive.d in the manual was in
> internals/data-structure. So the reason for this behaviour is that all of a sudden
> the whole repo became corrupted and borg had to rebuild all indices?

It's about cache coherency, not about corruption.

Of course, if your local index gets corrupt somehow and you need to kill
it, you will also have to rebuild it.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From f.dittmer at pyoworks.com  Fri May 29 08:07:07 2020
From: f.dittmer at pyoworks.com (Florian Dittmer)
Date: Fri, 29 May 2020 14:07:07 +0200
Subject: [Borgbackup] Rebuilding the cache - how does it work?
Message-ID: <2320384.XAFRqVoOGU@sapphire>

Hallo,

I find the following statement in the FAQ :

"If Borg detects that a repository has been modified since the local cache was 
updated it will need to rebuild the cache. This rebuild can be quite time 
consuming."

I have questions regarding this:

1. How exactly will Borg rebuild this cache?  Which data of the repository 
will Borg read and use to regenerate this cache? Only meta-data or the actual 
archived data? Will it re-use e.g. hash checksums from the archive's metadata 
or recalculate them on basis of actual data in the archive?

2. In a client/server setup, will the server or the client do the (if any) CPU 
intensive recalculation? Which (amount of) data will be transferred from 
server to client, in order to rebuild the cache? Only metadata , actual 
archive contents, or the server-side regenerated cache data?

3. In a client-only setup with archive storage on a mounted external network 
filesystem (e.g. sshfs), which (amount of) data will be transferred in order to 
rebuild the cache? Only metadata or actual archive contents?

Thank you very much

Florian





From volker.tanger at wyae.de  Fri May 29 19:30:46 2020
From: volker.tanger at wyae.de (Volker Tanger)
Date: Sat, 30 May 2020 01:30:46 +0200
Subject: [Borgbackup] borg create - with option bit-rot protection/detection
 for the original data
Message-ID: <20200530013046.79c68376@moira.wyae.de>

Hi!

By default new or changed files are detected by changed
ctime/size/inode - and only those are chunked and added 
to the archive. 

When you choose "rechunk" all files are freshly archived. 


One idea I had was to use the existing cache data and add 
new/changed files to the archive as detected by ctime/size/inode 
(as is standard behaviour) 
and rechunk all others but NOT write changed parts to the archive,
but instead issue a warning about possibly rotted bits on the
originating system.
After all, a file should not have changed (under normal circumstances)
if that is not also reflected in a change in ctime/size/inode.

The idea is to detect corrupt files that might be backed up otherwise
and overwrite/prune the "real" backup, e.g. when a new archive is
created or an old one is rechunked.


Is someone already working on such a functionality to "abuse" Borg as
bit-rot detection/protection for the client? 

Or would such a functionality be better implemented in other systems
(e.g. using/extending md5sum/sha256sum) - when you cannot use a file
system that already includes bit-rot protection (currently I only
know of ZFS, BTRFS and bcachefs to have checksums on data)?

Thanks

Volker


-- 

Volker Tanger    http://www.wyae.de/volker.tanger/
--------------------------------------------------
volker.tanger at wyae.de              PGP Fingerprint
5ED5 CE5E 1D3D 56F4 8990  70EA 3F04 530E 6D71 9D00

From tw at waldmann-edv.de  Sat May 30 06:48:25 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 30 May 2020 12:48:25 +0200
Subject: [Borgbackup] borg create - with option bit-rot
 protection/detection for the original data
In-Reply-To: <20200530013046.79c68376@moira.wyae.de>
References: <20200530013046.79c68376@moira.wyae.de>
Message-ID: <fa1789a6-68ca-04a6-b6e6-4b6930981491@waldmann-edv.de>

Interesting idea, but there are some issues:

1. Ideally, the checksum for some data should be computed when the data
is written to the filesystem and should be stored with the data
(filesystems like ZFS do that)

But borg does compute the checksums (chunk ids) when it reads the data
(to create a backup of it), so there is some delay and any bitrot in
that delay time would not be noticed by borg.

A user might even get the impression that everything is fine for sure
although the file was corrupted before borg has even seen it.

2. Also, ideally, bitrot should be detected (and fixed, if possible)
when reading the data. Filesystems like ZFS do that (when accessing the
data or when a regular scrub operation over the complete pool is done).

Borg can't do that and would only notice next time when you run it in
bitrot-detect mode.

3. The borg cache is also just a cache and information about files might
be removed (e.g. if the file was not seen in the last N backups).

4. Also, it is of course a rather expensive/slow operation, because all
files would be read / chunked / hashed completely.

5. The resulting lists of chunkids can only be used for comparison if
the chunker params are the same as at the time when the cache entry was
created (and we do not store the chunker params info in the cache entries).

So I guess users who care about filesystem bitrot are better advised by
using a filesystem with checksums and redundancy. When using a ZFS
mirror, ZFS can even fix bitrot if one side of the mirror still has a
valid copy of the data.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Sat Jun  6 09:50:10 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 6 Jun 2020 15:50:10 +0200
Subject: [Borgbackup] borgbackup 1.1.12 released!
Message-ID: <648463c2-366b-4af6-844b-c5be3670d0a8@waldmann-edv.de>


Including fixes for a major memory leak and improved FUSE read speed.

https://github.com/borgbackup/borg/releases/tag/1.1.12

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Sat Jun  6 18:52:26 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 7 Jun 2020 00:52:26 +0200
Subject: [Borgbackup] borgbackup 1.1.13 released!
Message-ID: <b1a11ca5-a44e-7779-92f8-fec99744186b@waldmann-edv.de>


Including fixes for a major memory leak and improved FUSE read speed.

https://github.com/borgbackup/borg/releases/tag/1.1.13

Same as 1.1.12, but cythonized with a py38 compatible cython.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From panayotis at panayotis.com  Mon Jun  8 10:03:32 2020
From: panayotis at panayotis.com (Panayotis Katsaloulis)
Date: Mon, 8 Jun 2020 17:03:32 +0300
Subject: [Borgbackup] backup from different machines with different user-ids
In-Reply-To: <bdcbc4a0-f1fc-4a76-a7ea-bb83b32f7758@Spark>
References: <bdcbc4a0-f1fc-4a76-a7ea-bb83b32f7758@Spark>
Message-ID: <e5561e34-4840-4fbd-8e9b-66eaa2348457@Spark>

Hello people

Today I have decided to update my borg server to the latest version. Up to now I used to make backup from different machines (with practically identical data), from my Linux and my macOS box.

The problem is on my Linux box, the user has ID 1000 while on the macOS the user has ID 501.
In the older versions of borg there wasn?t any problem with that, or at least any problem that would be obvious.
I was using??borg mount to see??and transfer files, and it used to work fine with rsync.

Unfortunately now this inconsistency results to a ?Permission denied? error.??I understand that this problem comes originally from this??inconsistency. My question is how to fix it? And what would you suggest as a best method to handle this situation?

As a side note, the directory which is backed up, belongs to ?me?, the same user with the same username on both machines.

--
Panayotis
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200608/0422f576/attachment.html>

From lazyvirus at gmx.com  Mon Jun  8 10:20:59 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Mon, 8 Jun 2020 16:20:59 +0200
Subject: [Borgbackup] backup from different machines with different
 user-ids
In-Reply-To: <e5561e34-4840-4fbd-8e9b-66eaa2348457@Spark>
References: <bdcbc4a0-f1fc-4a76-a7ea-bb83b32f7758@Spark>
 <e5561e34-4840-4fbd-8e9b-66eaa2348457@Spark>
Message-ID: <20200608162059.6257e152@msi.defcon1.lan>

On Mon, 8 Jun 2020 17:03:32 +0300
Panayotis Katsaloulis <panayotis at panayotis.com> wrote:

> Hello people

Hi Panayotis,

> Today I have decided to update my borg server to the latest version.
> Up to now I used to make backup from different machines (with
> practically identical data), from my Linux and my macOS box.
> 
> The problem is on my Linux box, the user has ID 1000 while on the
> macOS the user has ID 501. In the older versions of borg there wasn?t
> any problem with that, or at least any problem that would be obvious.
> I was using??borg mount to see??and transfer files, and it used to
> work fine with rsync.
> 
> Unfortunately now this inconsistency results to a ?Permission denied?
> error.??I understand that this problem comes originally from
> this??inconsistency. My question is how to fix it? And what would you
> suggest as a best method to handle this situation?

The fastest one I see is :
* mounting the backup on a server directory,
* export this directory from the server on NFS,
* mount this NFS share onto the client,
* client copy files from the mounted NFS share to the disk.

This way, the Id correspondance daemon will play its role and you
shouldn't have any discrepancy in (same) users numeric Ids.
 
> As a side note, the directory which is backed up, belongs to ?me?, the
> same user with the same username on both machines.

Jean-Yves

From panayotis at panayotis.com  Mon Jun  8 14:53:25 2020
From: panayotis at panayotis.com (Panayotis Katsaloulis)
Date: Mon, 8 Jun 2020 21:53:25 +0300
Subject: [Borgbackup] backup from different machines with different
 user-ids
In-Reply-To: <20200608162059.6257e152@msi.defcon1.lan>
References: <bdcbc4a0-f1fc-4a76-a7ea-bb83b32f7758@Spark>
 <e5561e34-4840-4fbd-8e9b-66eaa2348457@Spark>
 <20200608162059.6257e152@msi.defcon1.lan>
Message-ID: <e027984b-db6e-40bb-b285-dc04ee7d951e@Spark>

Hello and thank you for the reply!

My problem is not really how to retrieve these files, since I do have access ? even as root ? to these files, if/when I need them, I believe I could be able to retrieve them.

My question is (if I understand correctly) more about what you call "correspondence daemon?.
Or, how to match names and don?t worry about the different ids.
Is there any more info about it and how the match is performed?

--
Panayotis

--
Panayotis
On 8 Jun 2020, 5:21 PM +0300, Bzzzz <lazyvirus at gmx.com>, wrote:
> On Mon, 8 Jun 2020 17:03:32 +0300
> Panayotis Katsaloulis <panayotis at panayotis.com> wrote:
>
> > Hello people
>
> Hi Panayotis,
>
> > Today I have decided to update my borg server to the latest version.
> > Up to now I used to make backup from different machines (with
> > practically identical data), from my Linux and my macOS box.
> >
> > The problem is on my Linux box, the user has ID 1000 while on the
> > macOS the user has ID 501. In the older versions of borg there wasn?t
> > any problem with that, or at least any problem that would be obvious.
> > I was using??borg mount to see??and transfer files, and it used to
> > work fine with rsync.
> >
> > Unfortunately now this inconsistency results to a ?Permission denied?
> > error.??I understand that this problem comes originally from
> > this??inconsistency. My question is how to fix it? And what would you
> > suggest as a best method to handle this situation?
>
> The fastest one I see is :
> * mounting the backup on a server directory,
> * export this directory from the server on NFS,
> * mount this NFS share onto the client,
> * client copy files from the mounted NFS share to the disk.
>
> This way, the Id correspondance daemon will play its role and you
> shouldn't have any discrepancy in (same) users numeric Ids.
>
> > As a side note, the directory which is backed up, belongs to ?me?, the
> > same user with the same username on both machines.
>
> Jean-Yves
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200608/e2c5ab18/attachment.html>

From lazyvirus at gmx.com  Mon Jun  8 15:25:44 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Mon, 8 Jun 2020 21:25:44 +0200
Subject: [Borgbackup] backup from different machines with different
 user-ids
In-Reply-To: <e027984b-db6e-40bb-b285-dc04ee7d951e@Spark>
References: <bdcbc4a0-f1fc-4a76-a7ea-bb83b32f7758@Spark>
 <e5561e34-4840-4fbd-8e9b-66eaa2348457@Spark>
 <20200608162059.6257e152@msi.defcon1.lan>
 <e027984b-db6e-40bb-b285-dc04ee7d951e@Spark>
Message-ID: <20200608212544.600172f5@msi.defcon1.lan>

On Mon, 8 Jun 2020 21:53:25 +0300
Panayotis Katsaloulis <panayotis at panayotis.com> wrote:

> My problem is not really how to retrieve these files, since I do have
> access ? even as root ? to these files, if/when I need them, I believe
> I could be able to retrieve them.
> 
> My question is (if I understand correctly) more about what you call
> "correspondence daemon?. Or, how to match names and don?t worry about
> the different ids. Is there any more info about it and how the match
> is performed?

AFAI understand it, it pairs the user's numerical Id with his name,
thus, if user user1 at domain is 1000 on the server and 1005 on the client,
nfsidmap translate numerical Ids in a bidirectional way and the same for
the user's group Id.

Formerly, it was quite messy, as this daemon was out of any nfs package,
not using nfs in it's name (idmap IIRC) and wasn't even referenced as
suggested - today, you don't care, as it has been integrated into the
nfs-common package, meaning it is installed on either a server or a
client.

You can influence it by editing /etc/idmapd.conf (you can see that in
the default, it maps user nobody and group nogroup), but as I never used
it, I duno if it must be made on the server or on the client.
Note that, excepted for users/groups that do not have the exact same name
on the server and the client, the interest of this file is limited.

JY

From spython01 at gmail.com  Tue Jun  9 12:12:16 2020
From: spython01 at gmail.com (Samir Parikh)
Date: Tue, 9 Jun 2020 12:12:16 -0400
Subject: [Borgbackup] SSH Connection Broken Pipe Error
Message-ID: <CAMj1aXU8xs-UvTATWQ6najNPm5YmcubwgDddK_pj=jLX7Yomjw@mail.gmail.com>

Hi Everyone,

I am attempting to backup the files on my iMac running macOS Mojave
10.14.6 and borg version 1.1.11 to a repository on a virtual private
server running Ubuntu 18.04.4 and borg version 1.1.11.

I am getting the following error when backing up directories with a
large number of big files (videos, music, photos):

Remote: packet_write_wait: Connection to 123.45.67.890: Broken pipe
Connection closed by remote host

I am able to backup my Documents, Desktop and Download directories as
separate archives with no issue.  However, I am having problems
backing up my Movies folder which contains a lot of video files.  The
command I am using is:

/usr/local/bin/borg create                      \
        --verbose                               \
        --list                                  \
        --progress                              \
        --stats                                 \
        ::$NOW-samir-movies                     \
        /Users/samir/Movies                     \
        2>> "$LOGDIR/$NOW-log-samir-movies?

Per the instructions in issue 3988[1], I have set the ~/.ssh/config
file on my iMac (local machine) to:

Host 123.45.67.890
        ServerAliveInterval 10
        ServerAliveCountMax 30

and have the following configuration in the /etc/ssh/sshd_config file
on the server containing the remote repository:

ClientAliveInterval 10
ClientAliveCountMax 30

but I am still getting the "broken pipe" error which kills the backup.
Any ideas on what I can try to resolve this?

Thanks
Samir

[1] https://github.com/borgbackup/borg/issues/3988

From w at swtk.info  Wed Jun 17 13:37:13 2020
From: w at swtk.info (Wojtek Swiatek)
Date: Wed, 17 Jun 2020 19:37:13 +0200
Subject: [Borgbackup] Exception: /root/.cache/borg/c68...81e/config does not
 look like a Borg cache.
Message-ID: <CAPRYYOZ1g6W5-YxgGrA0h+uZ_ANb_Rz_j24pFpuNYYxEQE6Zew@mail.gmail.com>

Hello everyone

I realized that one of my backups is not working anymore. It is launched
together with another one which backs up to the same place (just a
different directory) though the same command - and that second one works
without problems.

When trying to list the contents of the failing backup (to know how late I
am) I see

/etc/local # borg list /services/backup/borg/srv
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4529, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 4461, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 157, in
wrapper
    assert_secure(repository, kwargs['manifest'], self.lock_wait)
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 206, in
assert_secure
    sm.assert_secure(manifest, manifest.key, lock_wait=lock_wait)
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 171, in
assert_secure
    with cache_config:
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 238, in
__enter__
    self.open()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 262, in open
    self.load()
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 268, in load
    self._check_upgrade(self.config_path)
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 327, in
_check_upgrade
    raise Exception('%s does not look like a Borg cache.' % config_path)
from None
Exception:
/root/.cache/borg/c6853ce826e2c5c62c58df2ac0eabca4cf087980944b57995703bd44cceb981e/config
does not look like a Borg cache.

Platform: Linux srv 5.3.0-59-generic #53~18.04.1-Ubuntu SMP Thu Jun 4
14:58:26 UTC 2020 x86_64
Linux: Ubuntu 18.04 bionic
Borg: 1.1.11  Python: CPython 3.6.9 msgpack: 0.5.6
PID: 23732  CWD: /etc/local
sys.argv: ['/usr/bin/borg', 'list', '/services/backup/borg/srv']
SSH_ORIGINAL_COMMAND: None


In contrast the working command:

/etc/local # borg list /services/backup/borg/edgerouter
b44277ee-4a84-4b73-b2db-69ea511be6e4 Sat, 2020-02-29 23:20:57
[822e95d0bbf6a8cbe5be6432e5044aea46d228ac1ed56474383a9daa594e0c69]
e0f38c38-f5aa-4531-aa2c-32bfdba0856f Tue, 2020-03-31 23:21:33
[306ff5ecf2f3995b82cfbf42607b6ccf904e473966ccf480e06fac20d5b236ae]
569ad25d-11a2-48e7-8c0c-513d45c88ea5 Thu, 2020-04-30 23:21:13
[30230caabb0ade5a77fde0776b61568f7f1ec3760502aed473eac53cabfe3d74]
ac3b8f7d-41c4-4add-aff2-88619d73c6c6 Sun, 2020-05-17 23:21:49
[cbd87b9d1cbfa2c7b9aa902d15f52773eb6eb6640bc8e45d140a30c6416e32fb]
(...)


Where should I go from there so that
/root/.cache/borg/c6853ce826e2c5c62c58df2ac0eabca4cf087980944b57995703bd44cceb981e/config
starts to look like a Borg cache? (or invalidate that specific backup).
I would be keen to retain the existing backups but if recovering is tricky
I will start from scratch to have something at least
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20200617/b4e37456/attachment.html>

From tw at waldmann-edv.de  Wed Jun 17 13:56:59 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Wed, 17 Jun 2020 19:56:59 +0200
Subject: [Borgbackup] Exception: /root/.cache/borg/c68...81e/config does
 not look like a Borg cache.
In-Reply-To: <CAPRYYOZ1g6W5-YxgGrA0h+uZ_ANb_Rz_j24pFpuNYYxEQE6Zew@mail.gmail.com>
References: <CAPRYYOZ1g6W5-YxgGrA0h+uZ_ANb_Rz_j24pFpuNYYxEQE6Zew@mail.gmail.com>
Message-ID: <8fd9cb1d-1f87-d818-ce84-940878f1fa77@waldmann-edv.de>

# borg list /services/backup/borg/srv
> Exception:
> /root/.cache/borg/c6853ce826e2c5c62c58df2ac0eabca4cf087980944b57995703bd44cceb981e/config
> does not look like a Borg cache.

If you're curious, have a look into that file it complains about. Guess
it is either empty or corrupted somehow.

The "fix" is getting rid of the corrupted cache:

borg delete --cache-only /services/backup/borg/srv

The next few operations after that will take a bit longer because it
will need to rebuild the misc. caches / indexes.


-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From lazyvirus at gmx.com  Tue Jun 30 00:19:21 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Tue, 30 Jun 2020 06:19:21 +0200
Subject: [Borgbackup] Repo init failure on a NFS directory with a raspberry
 pi 4
Message-ID: <20200630061921.571354a1@msi.defcon1.lan>

Hi borgbackupeters,

I installed 1.1.13 with pip3 without any problem, however the repo init
on a NFS share is systematically failing (I'm doing that on several x86
machines without any problem - NFS mount options are the same).

Here is the output of the init command :

$ sudo borg init -e keyfile /BORG/
Enter new passphrase:
Enter same passphrase again:
Do you want your passphrase to be displayed for verification? [yN]:
Fatal Python error: Bus error

Current thread 0xf79df8e0 (most recent call first):
  File
"/usr/local/lib/python3.7/dist-packages/borg/crypto/file_integrity.py",
line 75 in write
  File
"/usr/local/lib/python3.7/dist-packages/borg/crypto/file_integrity.py",
line 29 in write
  File
"/usr/local/lib/python3.7/dist-packages/borg/algorithms/msgpack/__init__.py",
line 38 in pack
  File
"/usr/local/lib/python3.7/dist-packages/borg/repository.py", line 613 in
write_index
  File
"/usr/local/lib/python3.7/dist-packages/borg/repository.py", line 476 in
commit File "/usr/local/lib/python3.7/dist-packages/borg/archiver.py",
line 297 in do_init
  File
"/usr/local/lib/python3.7/dist-packages/borg/archiver.py", line 176 in
wrapper File "/usr/local/lib/python3.7/dist-packages/borg/archiver.py",
line 4497 in run
  File
"/usr/local/lib/python3.7/dist-packages/borg/archiver.py", line 4565 in
main File "/usr/local/bin/borg", line 10 in <module>
Erreur du bus


It also fails with repokey.

Could somebody explain me why is that and what to do to make it work?

From alessandro.baggi at gmail.com  Tue Jun 30 05:18:03 2020
From: alessandro.baggi at gmail.com (Alessandro Baggi)
Date: Tue, 30 Jun 2020 11:18:03 +0200
Subject: [Borgbackup] borgbackup suggestion
Message-ID: <938dc7ad-6776-457b-ea89-0e1476d9f97f@gmail.com>

Hi everyone,

I'm new to borgbackup and I'm planning to switch my home backup server 
based on bacula to borg. Then after trying it extensively I would like 
to replace some backup solution with borg if all will go as intented.

Why migrate from bacula: I used bacula for many years (> 5) and I found 
always the volume managing, pooling, recycling too much complicated and 
(in some way) not so robust when I need to run backup on disk and not on 
TAPE. Over this there is also problem like adding a new client, database 
cleaning, deduplication not available without pain, encrypted connection 
and encrypted archive. After several years I noticed that I need 
something simpler and pratical.

Currently in my LAN I need to backup one workstation (Linux Based), a 
NAS (Linux based), a virtual machine (data not raw image) and a remote VPS.

I have some questions:

1) About encryption what is more secure between repokey and keyfile?
2) About managing borg as a central solution. I explain. Using bacula I 
can run backup and restore from my server for a specified client. I 
noticed that with borg I need to connect to the client and run command 
for restore, run backup, run prune, search file, list backups, run check 
and so on. For operation like backup, prune and check cron could help 
(but this implies that I need to configure for each client a crontab) 
but for other operation I need a shell to the client. There is a way to 
perform all this operations from a single host? If this is not the right 
way to work with borgbackup I am open to any suggestion. I ask this 
because in the future I could have many machines to backup and a central 
way could save time.

3) About pruning repository with --append-only enabled. I read on a 
reddit post that with --append-only mode I can prune old jobs on repos 
from a trusted host and not run the prune command from the client but I 
don't understand how to do this when all information are stored on the 
client. So How I can prune a specific repository from a trusted machine?

Thank you in advance.


From tw at waldmann-edv.de  Tue Jun 30 09:40:53 2020
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Tue, 30 Jun 2020 15:40:53 +0200
Subject: [Borgbackup] Repo init failure on a NFS directory with a
 raspberry pi 4
In-Reply-To: <20200630061921.571354a1@msi.defcon1.lan>
References: <20200630061921.571354a1@msi.defcon1.lan>
Message-ID: <dd8530e4-98fe-c0d1-ef6b-26bb371c0629@waldmann-edv.de>

> "/usr/local/lib/python3.7/dist-packages/borg/crypto/file_integrity.py",
> line 75 in write
> Bus Error

That is likely an unaligned access.

This occurs when not running ARM in "fixup" mode, then it just blows up 
when doing an unaligned access.

In "fixup" mode, a software exception handler would just make the access 
work as expected.

I heard that on raspi, the 32bit kernel usually runs in fixup mode, but 
the 64bit kernel does not (which i would say is an obviously bad idea).

IIRC:

- there is an issue about that in our github issue tracker
- the unaligned access is not in borg code, but in a library / 3rd party 
code borg uses


-- 

GPG Fingerprint: 6D5B EF9A DD20 7580 5747  B70F 9F88 FB52 FAF7 B393
Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt.

From lazyvirus at gmx.com  Tue Jun 30 09:54:51 2020
From: lazyvirus at gmx.com (Bzzzz)
Date: Tue, 30 Jun 2020 15:54:51 +0200
Subject: [Borgbackup] Repo init failure on a NFS directory with a
 raspberry pi 4
In-Reply-To: <dd8530e4-98fe-c0d1-ef6b-26bb371c0629@waldmann-edv.de>
References: <20200630061921.571354a1@msi.defcon1.lan>
 <dd8530e4-98fe-c0d1-ef6b-26bb371c0629@waldmann-edv.de>
Message-ID: <20200630155451.11f13b67@msi.defcon1.lan>

On Tue, 30 Jun 2020 15:40:53 +0200
Thomas Waldmann <tw at waldmann-edv.de> wrote:

> > "/usr/local/lib/python3.7/dist-packages/borg/crypto/file_integrity.py",
> > line 75 in write
> > Bus Error
>
> That is likely an unaligned access.
>
> This occurs when not running ARM in "fixup" mode, then it just blows
> up when doing an unaligned access.

What do you call an unaligned access? Do you speak about memory
alignment on CPU word width?

> In "fixup" mode, a software exception handler would just make the
> access work as expected.
>
> I heard that on raspi, the 32bit kernel usually runs in fixup mode,
> but the 64bit kernel does not (which i would say is an obviously bad
> idea).
>
> IIRC:
>
> - there is an issue about that in our github issue tracker

Ok, I wasn't sure it was exactly the same issue.

> - the unaligned access is not in borg code, but in a library / 3rd
> party code borg uses

I go back to read the whole ticket.

JY