From cherio at gmail.com  Fri Jul  2 15:26:44 2021
From: cherio at gmail.com (Cherio)
Date: Fri, 2 Jul 2021 15:26:44 -0400
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
Message-ID: <CAKHqFkJsBYpJ6vqBYdXZ4b80WbKLrzsqb8HdV+0jd3Z0pEcUsA@mail.gmail.com>

I'm feeling a bit awkward asking again.
Is this the right place to ask questions like this?

I would greatly appreciate a tip from knowledgeable people. I just want to
underscore that SSH request doesn't even leave the local machine, the
version mismatch occurs somewhere locally.

On Sun, Jun 27, 2021 at 11:05 PM Cherio <cherio at gmail.com> wrote:

> At some point (I can pinpoint exactly when) borg stopped working with
> remote targets via ssh. The requests don't even seem to be leaving the
> machine that starts the backup and halts with a message "Remote: OpenSSL
> version mismatch. Built against 1010106f, you have 101000cf". The request
> doesn't seem to be even reaching the remote backup machine.
>
> I run borg 1.1.15 on Ubuntu 20.04. I tried standalone binaries both from
> borg-linux64.tgz and as a single binary borg-linux64 with the same result.
> I also tried downloading 1.1.16 and it was having the exact same issue.
>
> Below are some version tests. I can't figure out where OpenSSL version
> 101000cf comes from.
>
> > openssl version
> OpenSSL 1.1.1f  31 Mar 2020
>
> > ssh -V
> OpenSSH_8.2p1 Ubuntu-4ubuntu0.2, OpenSSL 1.1.1f  31 Mar 2020
>
> > python
> Python 3.8.5 (default, May 27 2021, 13:30:53)
> [GCC 9.3.0] on linux
> >>> import ssl
> >>> ssl.OPENSSL_VERSION
> 'OpenSSL 1.1.1f  31 Mar 2020'
>
> I am hoping someone here may have an immediate light bulb and point me in
> the right direction.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210702/ad77f3cc/attachment.html>

From jolson at kth.se  Fri Jul  2 15:54:31 2021
From: jolson at kth.se (Jonas Olson)
Date: Fri, 2 Jul 2021 21:54:31 +0200
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
Message-ID: <ae5eb2f8-6158-baed-67ce-7ac91e792e46@kth.se>

On?2021-06-28?05:05,?Cherio?wrote:
> At some point (I can pinpoint exactly when) borg stopped working with 
> remote targets via ssh. The requests don't even seem to be leaving the 
> machine that starts the backup and halts with a message "Remote: 
> OpenSSL version mismatch. Built against 1010106f, you have 101000cf". 
> The?request?doesn't?seem?to?be?even?reaching?the?remote?backup?machine.

It looks like [0] this is a message from SSH and that the version 
mismatch exists between SSH and OpenSSL. Can you reproduce the error 
using?SSH?(without?Borg),?or?maybe?reinstall?as?suggested?in?the?link?

Jonas?Olson

[0] 
<https://stackoverflow.com/questions/24096069/openssl-version-mismatch-built-against-1000105f-you-have-10001080> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210702/7aaf5b50/attachment.html>

From tw at waldmann-edv.de  Sat Jul  3 08:40:37 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 3 Jul 2021 14:40:37 +0200
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
Message-ID: <2d6cab15-93d1-3fbb-9a39-3035b3aecc04@waldmann-edv.de>

This looks similar to this issue, which should be fixed since long:

https://github.com/borgbackup/borg/issues/514


-- 
GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Sat Jul  3 08:56:57 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 3 Jul 2021 14:56:57 +0200
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
Message-ID: <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>

Maybe there is a bug in either pyinstaller or in that code:

https://github.com/borgbackup/borg/blob/1.1.16/src/borg/helpers.py#L2407

Depending on the borg release, I used different pyinstaller versions:

1.1.16: pyinstaller 4.2
1.1.15: pyinstaller 4.0
older: pyinstaller 3.2.1

So, as you've already tried 1.1.15 and 1.1.16, it would be good to also 
try 1.1.14 or older, to see if there is a regression in recent 
pyinstaller versions.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From cherio at gmail.com  Sat Jul  3 19:59:19 2021
From: cherio at gmail.com (Cherio)
Date: Sat, 3 Jul 2021 19:59:19 -0400
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
 <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
Message-ID: <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>

SSH to anywhere has been working flawlessly (my daily routine relies on it)
so mismatch between SSH and OpenSSL should be ruled out.

I am using borg-linux64.tgz version, not the pyinstaller.
Surprisingly switching back to borg 1.1.14 (borg-linux64.tgz) makes
everything work again, thanks for the idea! Both 1.1.15 and 1.1.16 from
borg-linux64.tgz are a no-go for remote SSH sync for me. I noticed the
contents of the root directory in the latter 2 versions changed a lot. I
wonder if one of the supplied *.so libraries introduces a conflict.

I am very surprised I am the only one having the version mismatch issue.


On Sat, Jul 3, 2021 at 8:57 AM Thomas Waldmann <tw at waldmann-edv.de> wrote:

> Maybe there is a bug in either pyinstaller or in that code:
>
> https://github.com/borgbackup/borg/blob/1.1.16/src/borg/helpers.py#L2407
>
> Depending on the borg release, I used different pyinstaller versions:
>
> 1.1.16: pyinstaller 4.2
> 1.1.15: pyinstaller 4.0
> older: pyinstaller 3.2.1
>
> So, as you've already tried 1.1.15 and 1.1.16, it would be good to also
> try 1.1.14 or older, to see if there is a regression in recent
> pyinstaller versions.
>
> --
>
> GPG ID: 9F88FB52FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210703/ee569d83/attachment.html>

From tw at waldmann-edv.de  Sun Jul  4 04:08:15 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 4 Jul 2021 10:08:15 +0200
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
 <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
 <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>
Message-ID: <a1bbfdbe-3623-ad01-467d-dbe7392f51f7@waldmann-edv.de>

> I am using borg-linux64.tgz version, not the pyinstaller.

Both the borg-linux64 and borg-linux64.tgz binary releases are made with 
pyinstaller - the first is the single file bundle, the latter is the 
single directory bundle.

Can you try the single-file bundle also for 1.1.14 and 1.1.16?

> Surprisingly switching back to borg 1.1.14 (borg-linux64.tgz) makes 
> everything work again, thanks for the idea!

OK, so it seems we have a bug or regression either in borg, borg pyi 
building or pyinstaller 3.2.1 -> 4.0/4.2. :-(

> Both 1.1.15 and 1.1.16 from 
> borg-linux64.tgz are a no-go for remote SSH sync for me. I noticed the 
> contents of the root directory in the?latter 2 versions changed a lot. I 
> wonder if one of the supplied *.so libraries introduces?a conflict.

Our code to remove the pyinstaller LD_LIBRARY_PATH should make sure that 
your system ssh is loading your system libs (not the pyinstaller bundled 
libs).

> I am very surprised?I am the only one having the version mismatch issue.

At least you're the first one reporting it (after 2016).

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Sun Jul  4 04:24:16 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 4 Jul 2021 10:24:16 +0200
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <a1bbfdbe-3623-ad01-467d-dbe7392f51f7@waldmann-edv.de>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
 <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
 <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>
 <a1bbfdbe-3623-ad01-467d-dbe7392f51f7@waldmann-edv.de>
Message-ID: <485b1a15-b625-1d72-0802-1f0bb1a8a79f@waldmann-edv.de>

> Can you try the single-file bundle also for 1.1.14 and 1.1.16?

Ignore that, I've read in your first post that you have seen the bug 
also in the single-file bundle.

I've opened an issue there, summarizing your finding and the replies:

https://github.com/borgbackup/borg/issues/5897

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Sun Jul  4 05:23:20 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 4 Jul 2021 11:23:20 +0200
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <485b1a15-b625-1d72-0802-1f0bb1a8a79f@waldmann-edv.de>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
 <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
 <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>
 <a1bbfdbe-3623-ad01-467d-dbe7392f51f7@waldmann-edv.de>
 <485b1a15-b625-1d72-0802-1f0bb1a8a79f@waldmann-edv.de>
Message-ID: <2c62cf95-4ebc-c78d-922d-9acc6aa5def4@waldmann-edv.de>

>> Can you try the single-file bundle also for 1.1.14 and 1.1.16?
> 
> Ignore that, I've read in your first post that you have seen the bug 
> also in the single-file bundle.

Hmm, i did some experiments and think that:
- 1.1.16 single-dir pyinstaller bundle has that bug
- 1.1.16 single-file pyinstaller bundle should work, no bug

@cherio can you try again please and confirm?

> https://github.com/borgbackup/borg/issues/5897

If you have a github account, we could also continue there.

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From cherio at gmail.com  Sun Jul  4 15:28:12 2021
From: cherio at gmail.com (Cherio)
Date: Sun, 4 Jul 2021 15:28:12 -0400
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <2c62cf95-4ebc-c78d-922d-9acc6aa5def4@waldmann-edv.de>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
 <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
 <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>
 <a1bbfdbe-3623-ad01-467d-dbe7392f51f7@waldmann-edv.de>
 <485b1a15-b625-1d72-0802-1f0bb1a8a79f@waldmann-edv.de>
 <2c62cf95-4ebc-c78d-922d-9acc6aa5def4@waldmann-edv.de>
Message-ID: <CAKHqFkJEhKvNDUFAwnxHEG2_U47bxLm0ND+G11=8T-_kdCxRHw@mail.gmail.com>

Your suspicion is correct!!!!!!!

The single file 1.1.16 linux executable works. The directory
extracted borg-linux64.tgz doesn't.

On Sun, Jul 4, 2021 at 5:23 AM Thomas Waldmann <tw at waldmann-edv.de> wrote:

> >> Can you try the single-file bundle also for 1.1.14 and 1.1.16?
> >
> > Ignore that, I've read in your first post that you have seen the bug
> > also in the single-file bundle.
>
> Hmm, i did some experiments and think that:
> - 1.1.16 single-dir pyinstaller bundle has that bug
> - 1.1.16 single-file pyinstaller bundle should work, no bug
>
> @cherio can you try again please and confirm?
>
> > https://github.com/borgbackup/borg/issues/5897
>
> If you have a github account, we could also continue there.
>
> --
>
> GPG ID: 9F88FB52FAF7B393
> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210704/517c65d1/attachment.html>

From tschoening at am-soft.de  Tue Jul  6 16:14:41 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Tue, 6 Jul 2021 22:14:41 +0200
Subject: [Borgbackup] Borg client vs. server vs. NAS-like Borg server
Message-ID: <314524072.20210706221441@am-soft.de>

Hi all,

I have one server hosting VMs and one NAS-like device for backup
purposes providing necessary storage. That device exposes SSH with
RSYNC and especially Borg. As I want encrypted backups, Borg seems
superior over SSHFS+ECRYPTFS for the backup storage.

I would like to have a setup where I have one server pulling backups
from multiple different VMs, as I did that in the past with a NAS, SSH
and RSYNC. Additionally, this way I hope only one instance of Borg
needs to have access to all repos, their encryption keys and
passphrases. Regarding your FAQ, encryption is handled "client-side":

> The Borg server is doing storage-related low-level repo operations
> (get, put, commit, check, compact), while the Borg client does the
> high-level stuff: deduplication, encryption, compression, dealing
> with archives, backups, restores, etc., which reduces the amount of
> data that goes over the network.

https://borgbackup.readthedocs.io/en/stable/faq.html?what-is-the-difference-between-a-repo-on-an-external-hard-drive-vs-repo-on-a-server#what-is-the-difference-between-a-repo-on-an-external-hard-drive-vs-repo-on-a-server

In my setup, the Borg server would be the one running on the NAS-like
device providing backup storage, correct? The Borg client would then
be the one running on the VM-host backing up individual VMs.

Your docs describe two setups for PULL: SSHFS and ssh-agent. The
former has problems with unstable inodes, while the latter already
involves two instances of Borg as client and server. Additionally, I'm
not able to run arbitrary code on the backup storage.

So, with ssh-agent, the server would need to be the one of the
NAS-like storrage device, correct? In which case the client would need
to run within VMs, which would force me to make passphrases etc.
available to VMs, which I would like to avoid.

Or is there some way to make the Borg running in the VM client speak to
the Borg running on the VM host so that is acting as the client
compared to the Borg server on the NAS-like device?

Besides unstable inodes, does SSHFS imply any further restrictions
like not being able to backup hard-links? I've read one create those,
but they are handled as different files when queried.

Are there any other easy to implement, SSH-foccussed PULL-setups?
Something more like SSH+RSYNC used in the past? Some host logging into
some other, executing Borg to transfer files using the already
existing connection instead of creating a new one like in the setup
using "ssh-agent"?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Tue Jul  6 16:24:22 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Tue, 6 Jul 2021 22:24:22 +0200
Subject: [Borgbackup] borgmatic to backup databases
Message-ID: <1497112518.20210706222422@am-soft.de>

Hi all,

I know it's a different project, but maybe you have experience with
that as well: borgmatic as frontend to Borg looks pretty interesting
because it provides hooks and supports dumping PostgreSQL databases. I
need the former to create file system snapshots in ZFS and BTRFS if
available and the latter simply because some VMs use Postgres.

The important point is that I would like to avoid dumping Postgres
into the file system and instead would like to directly stream to the
Borg repo. borgmatic is able to do this in theory:

> As part of each backup, borgmatic streams a database dump for each
> configured database directly to Borg, so it's included in the backup
> without consuming additional disk space.

https://torsion.org/borgmatic/docs/how-to/backup-your-databases/

But here is where things become interesting: I would additionally like
to setup only one server backing up all VMs using a PULL-approach.
For files this might be possible using SSHFS or ...

But to support borgmatic and Postgres I would need to make Postgres
available to that host on network level as well, correct? This is
because borgmatic being the backup frontend is executed on the one
server intended to backup all other hosts. Or is it spawning
additional processes within the hosts to be backed up, like RSYNC over
SSH?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Tue Jul  6 13:54:47 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Tue, 6 Jul 2021 19:54:47 +0200
Subject: [Borgbackup] Suggested repo-layout: multiple vs. one
Message-ID: <1499156247.20210706195447@am-soft.de>

Hi all,

I'm new to Borg and implemented a PULL-approach in the past using some
Synology NAS and RSYNC: The NAS executed SSH+RSYNC to fetch some host
and store locally. I would like to do pretty much the same with Borg.

My storage consisted of two levels at the NAS: One large volume
formatted with BTRFS and within that each host or backup purpose was
one additional subvolume. The only relevant features used were
compression and snapshots and that way snapshots could be maintained
per subvolume/hst/backup purpose.

I have mostly three different types of data to backup:

* one host with VM images, mostly QCOW2 files for QEMU
* contents of the same VMs, being some Ubuntu Linux mostly
* dumps for Postgres created by pg_dump

Especially with looking at Borg's dedup capabilities, to me it makes
sense to create three individual repos for each of the above data.
This way all the data with the same characteristics is shared within
the same repo, while the concept of an ARCHIVE can be used to address
individual VMs or alike within that repo.

I don't think that too many chunks will be shared between image files
and VM-contents, so splitting up the repos reduces the chance of total
data loss in case anything goes wrong on repo level as well. Besides
that, I wouldn't like to encode too many things in the ARCHIVE name.

Is that a reasonable approach to you? Or am I missing something and
have an additional layer per REPO besides the ARCHIVE anyway?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Tue Jul  6 15:35:58 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Tue, 6 Jul 2021 21:35:58 +0200
Subject: [Borgbackup] PULL+SSHFS: Avoid CHROOT with --numeric-owner?
Message-ID: <774305292.20210706213558@am-soft.de>

Hi all,

one example in the docs uses SSHFS to make the client to be backed up
available to some server and CHROOT because of UID/GID mapping. I'm
aware already of INODE-related problems, but though would like to
understand that setup.

https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html?highlight=pull#creating-a-backup

The point is that the whole paragraph reads like its absolutely
necessary to do the CHROOT-stuff, while I'm not sure if I really need
names of users and groups at all. RSYNC only stored IDs in the past as
well and looking at your data structures, you seem to be able to store
IDs. Though, it's not clear to me if IDs only can be stored as well or
names are always needed. The docs don't say those fields are optional
or alike:

> user
> group
> uid
> gid

https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#items

OTOH, I've found the argument "--numeric-owner" to "borg create",
which exactly seems to imply that user and group names are optional.

> --numeric-owner only store numeric user and group identifiers

https://borgbackup.readthedocs.io/en/stable/usage/create.html

So, is the CHROOT-stuff really only needed for that mapping or is it
serving an additional purpose I simply didn't understand yet? In the
first case, can CHROOT be skipped entirely by simply creating backups
using "borg create --numeric-owner [...]"? Things sound easier to
setup when not needing to care about CHROOT.

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Tue Jul  6 15:37:41 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Tue, 6 Jul 2021 21:37:41 +0200
Subject: [Borgbackup] How to store keyfiles outside repo AND home directory?
Message-ID: <937660494.20210706213741@am-soft.de>

Hi all,

I have one server hosting VMs and one NAS-like device for backup
purposes. Though, it's not a NAS I can execute arbitrary code on,
things are limited to RSYNC and BORG, otherwise it's only providing
plain storage using SMB, WebDAV etc. It's somewhat important to have
encrypted backups, hence looking at Borg. Additionally, the provider
suggested it and having DEDUP+COMPRESS is superiour over e.g.
SSHF+ECRYPTFS.

Of course backups should be done regularly automatically, without me
providing some passphrase manually. Additionally, I would like to
avoid to store that unencrypted in files. I've already read about
special environment variables and it seems BORG_PASSCOMMAND is exactly
what I need: The server uses encrypted ZFS which will manually be
mounted and can easily store the passphrase for borg this way. This is
secure enough and better than storing in clear text at all. 

> BORG_PASSPHRASE
> BORG_PASSCOMMAND

https://borgbackup.readthedocs.io/en/stable/faq.html?highlight=borg_passcommand#how-can-i-specify-the-encryption-passphrase-programmatically

But here's the point: I'm as well considering using encryption mode
"keyfile", which by default seems to store those files in the
unencrpyted user home directory executing Borg.

https://borgbackup.readthedocs.io/en/stable/usage/init.html#
https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html#key-files

I know it's protected by the passphrase, but when storing outside the
repo, it makes sense to me to protect it the same way like the
passphrase. In my case that would mean storing it within the
enncrypted ZFS dataset used for the passphrase as well already.

So, is there some config I've missed to provide where to store the key
files or do I need to work with symlinks/mounts/... to move them out
of their default location?

Are there any known issues in case of using symlinks or else, like
problems with atomic file operations crossing file system borders?

How are the actual files named in "$HOME/.config/borg/keys"? I need to
support multiple different repos.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Tue Jul  6 15:39:05 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Tue, 6 Jul 2021 21:39:05 +0200
Subject: [Borgbackup] Did anyone ever test SSHFS with FUSES's
 NOFORGET-option?
Message-ID: <103460207.20210706213905@am-soft.de>

Hi all,

I'm refering to the following problem:

https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me

After looking at your issue tracker and stuff, whenever people ran
into that problem and SSHFS really was the issue, instead of changing
mount points or alike, I couldn't find the options people used SSHFS
with.

The point is that FUSE supports two interesting options: "use_ino",
which is documented to allow the file system to "set" inodes. OTOH,
I've read that SSHFS is based on SFTP, which doesn't provide inodes at
all. So "set" might really be meant as it is, like a value not being
available when only read in this setup.

> -o use_ino
> let filesystem set inode numbers

https://linux.die.net/man/1/sshfs

But there's additionally the option "noforget" and that doesn't seem
to even be documented for SSHFS:

> Never forget cached inodes

https://www.fsl.cs.stonybrook.edu/docs/fuse/fuse-article-appendices.html

And here's the interesting part: I've found someone describing inode
related problems with GIT and SSHFS and "noforget" seems to have fixed
those:

> Fuse version 2.8.x has a "noforget" option that should provide stable
> inode numbers, at the cost of unbounded memory use. Could you please
> try if this option fixes these issues?

https://git.vger.kernel.narkive.com/L4f8hTjT/inode-problem-when-using-git-on-a-sshfs-filesystem#post3
https://git.vger.kernel.narkive.com/L4f8hTjT/inode-problem-when-using-git-on-a-sshfs-filesystem#post4

As both mentioned options are disabled by default accoridng the docs,
I wondered if it's known that those have been explicitly tested ever?

Or is it known already that those can't work with Borg?

Especially the latter reads to me like FUSE is calculating inodes
somehow on its own and those are stable then onyl for a longer period
of time. But I couldn't find how those are calculated especially with
SSHFS and how long those would be valid. I guess only for how long a
file system is mounted?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From lazyvirus at gmx.com  Tue Jul  6 17:28:14 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Tue, 6 Jul 2021 23:28:14 +0200
Subject: [Borgbackup] Did anyone ever test SSHFS with FUSES's
 NOFORGET-option?
In-Reply-To: <103460207.20210706213905@am-soft.de>
References: <103460207.20210706213905@am-soft.de>
Message-ID: <20210706232814.0de396a3@msi.defcon1.lan>

On Tue, 6 Jul 2021 21:39:05 +0200
Thorsten Sch?ning<tschoening at am-soft.de> wrote:

> https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me
> 
> After looking at your issue tracker and stuff, whenever people ran
> into that problem and SSHFS really was the issue, instead of changing
> mount points or alike, I couldn't find the options people used SSHFS
> with.
> 
> The point is that FUSE supports two interesting options: "use_ino",
> which is documented to allow the file system to "set" inodes. OTOH,
> I've read that SSHFS is based on SFTP, which doesn't provide inodes at
> all. So "set" might really be meant as it is, like a value not being
> available when only read in this setup.
> 
> > -o use_ino
> > let filesystem set inode numbers
> 
> https://linux.die.net/man/1/sshfs
> 
> But there's additionally the option "noforget" and that doesn't seem
> to even be documented for SSHFS:
> 
> > Never forget cached inodes
> 
> https://www.fsl.cs.stonybrook.edu/docs/fuse/fuse-article-appendices.html
> 
> And here's the interesting part: I've found someone describing inode
> related problems with GIT and SSHFS and "noforget" seems to have fixed
> those:
> 
> > Fuse version 2.8.x has a "noforget" option that should provide stable
> > inode numbers, at the cost of unbounded memory use. Could you please
> > try if this option fixes these issues?
> 
> https://git.vger.kernel.narkive.com/L4f8hTjT/inode-problem-when-using-git-on-a-sshfs-filesystem#post3
> https://git.vger.kernel.narkive.com/L4f8hTjT/inode-problem-when-using-git-on-a-sshfs-filesystem#post4
> 
> As both mentioned options are disabled by default accoridng the docs,
> I wondered if it's known that those have been explicitly tested ever?
> 
> Or is it known already that those can't work with Borg?
> 
> Especially the latter reads to me like FUSE is calculating inodes
> somehow on its own and those are stable then onyl for a longer period
> of time. But I couldn't find how those are calculated especially with
> SSHFS and how long those would be valid. I guess only for how long a
> file system is mounted?

Type: borg help create
and look for the "--files-cache" section.

Jean-Yves

From tschoening at am-soft.de  Wed Jul  7 03:00:16 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Wed, 7 Jul 2021 09:00:16 +0200
Subject: [Borgbackup] Does anyone backup VMs using private Linux VM-bridges
 with SSHFS?
Message-ID: <131618375.20210707090016@am-soft.de>

Hi all,

I have Proxmox running multiple different VMs and am considering
backing those up using Borg with SSHFS. The goal is to have backup
managed by the Proxmox-host only using PULL, like is pretty common
with SSH+RSYNC.

Though, that setup is mentioned in the FAQs to be somewhat slow,
because a lot of data needs to pass SSH, while with a client-server
setup the client within the VM only sends actually changed chunks.

OTOH, I can easily create private bridges in Proxmox not bridged to
any physical interface, but implementing a private network for the VMs
and the host only. this could easily be a private backup network
this way, not creating any public traffic at all in theory.

https://forum.proxmox.com/attachments/xproxmox-root-server-architektur-1024x595-png-pagespeed-ic-nq1avqgasy-png.27234/
https://forum.proxmox.com/threads/whats-the-preferred-counterpart-for-internalnetwork-of-virtualbox.91686/#post-399961

From my understanding that would only keep the CPU overhead because of
SSH, but without too much network related latency, without bandwidth
limits of real physical NICs etc.


Or am I missing something? Does anyone has any experience with that
approach?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Wed Jul  7 03:54:40 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Wed, 7 Jul 2021 09:54:40 +0200
Subject: [Borgbackup] What are host and user name in archives used for?
Message-ID: <1647641287.20210707095440@am-soft.de>

Hi all,

from my understanding of Borg, in Borg itself I only have two levels
to organize my backups: The repo itself and multiple archives within.
Sharing a repo increases the chance to deduplicate data amongst all
contained archives, while each archive is something self contained,
like a host or a backup purpose or alike.

OTOH, those archives are associated with some point in time as well,
because each archive is one concrete backup. The latter means that one
needs to find some naming scheme to distinguish all those individual
archives, because in the end, depending on what gets backed up how
often, one might get a pretty long list like the following:

> repo::host-1_$TIMESTAMP-1
> [...]
> repo::host-N_$TIMESTAMP-N
> repo::other_purpose-1_$TIMESTAMP-1
> [...]
> repo::other_purpose-N_$TIMESTAMP-N

It's not clear to me currently if it's really only about a naming
convention or if there's additional metadata one can provide and is
used under some circumstances. For example, the docs regarding the
file format for an archive describe fields for some host and user:

> hostname
> username

https://borgbackup.readthedocs.io/en/stable/internals/data-structures.html?archives#archives

Additionally there's the env variable "BORG_HOST_ID" mentioning some
automatically done calculations. But I didn't find yet which instance
of Borg does fill those fields per archive under which setup: There's
a client-server-setup with possibly different hosts and users creating
the archive, e.g. compared to when using a PULL setup using SSHFS to
mount a remote client to be backed up.

https://borgbackup.readthedocs.io/en/stable/usage/general.html?#environment-variables
https://borgbackup.readthedocs.io/en/stable/faq.html#what-is-the-difference-between-a-repo-on-an-external-hard-drive-vs-repo-on-a-server
https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html

In the end, it's about e.g. pruning old archives, which provides the
argument "--prefix". From my understandig this really only applies to
the name of the archive, so I need to encode things like different
backed up hosts, purposes etc. really only there or afterwards I won't
be able to distinguish things when pruning anymore. E.g. when only
using generic names with timestamps, I wouldn't be able to prune
VM-contents more often than other backup purposes.

Or are there mechanisms that e.g. additionally filter by fields
"hostname", "username" etc. maintained by the archive itself? Maybe
only some commands do so and prune does not?

That might allow an additional level of distinction of archives in the
end, so that one might create backups by different users or alike.
Currently I'm considering to use one and the same user to exec only
one instance of Borg in PULL mode using SSHFS instead.

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From heiko.helmle at horiba.com  Wed Jul  7 04:34:36 2021
From: heiko.helmle at horiba.com (Heiko Helmle)
Date: Wed, 7 Jul 2021 08:34:36 +0000
Subject: [Borgbackup] Does anyone backup VMs using private Linux
 VM-bridges with SSHFS?
In-Reply-To: <131618375.20210707090016@am-soft.de>
References: <131618375.20210707090016@am-soft.de>
Message-ID: <TYAPR01MB631321D39C998C2FAB91B2C8E11A9@TYAPR01MB6313.jpnprd01.prod.outlook.com>

Hi Thorsten,

from my experience with proxmox and borg in particular:

your biggest challenge is probably the large amount of data to parse by borg on every backup (basic proxmox backups are always-full), and the fact that borg is single-threaded. This blew up our single-job backup process to last about 10 hours every day. As it only takes up a single CPU, impact is minimal, but your backup process taking that long might be unexpected.

So if you want to go borg, consider running several jobs in parallel (to different repos then). I'd seriously look into proxmox backup server though, as it allows proxmox to do true incremental backups and greatly decreases the amount of data that gets moved around.

Also be aware that the default chunker settings result in almost no deduplication with the proxmox archive format. I had good results with --chunker-params 16,23,16,4095, but you definitely need a few test runs to find the values that fit your workload.

Best Regards
 Heiko

-----Original Message-----
From: Borgbackup <borgbackup-bounces+heiko.helmle=horiba.com at python.org> On Behalf Of Thorsten Sch?ning
Sent: Mittwoch, 7. Juli 2021 09:00
To: borgbackup at python.org
Subject: [Borgbackup] Does anyone backup VMs using private Linux VM-bridges with SSHFS?

Hi all,

I have Proxmox running multiple different VMs and am considering backing those up using Borg with SSHFS. The goal is to have backup managed by the Proxmox-host only using PULL, like is pretty common with SSH+RSYNC.

Though, that setup is mentioned in the FAQs to be somewhat slow, because a lot of data needs to pass SSH, while with a client-server setup the client within the VM only sends actually changed chunks.

OTOH, I can easily create private bridges in Proxmox not bridged to any physical interface, but implementing a private network for the VMs and the host only. this could easily be a private backup network this way, not creating any public traffic at all in theory.

https://forum.proxmox.com/attachments/xproxmox-root-server-architektur-1024x595-png-pagespeed-ic-nq1avqgasy-png.27234/
https://forum.proxmox.com/threads/whats-the-preferred-counterpart-for-internalnetwork-of-virtualbox.91686/#post-399961

From my understanding that would only keep the CPU overhead because of SSH, but without too much network related latency, without bandwidth limits of real physical NICs etc.


Or am I missing something? Does anyone has any experience with that approach?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

--
AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil:
Webseite: https://www.am-soft.de

AM-Soft IT-Service - Bitstore Hameln GmbH i.G. ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service - Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491 876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH G?tersloh
Tel: 05245 920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln Gesch?ftsf?hrer Janine Galonska






_______________________________________________
Borgbackup mailing list
Borgbackup at python.org
https://mail.python.org/mailman/listinfo/borgbackup
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Gesch?ftsf?hrer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Takashi Nagano, Junichi Tajika, Ergin Cansiz.

From tschoening at am-soft.de  Wed Jul  7 05:57:01 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Wed, 7 Jul 2021 11:57:01 +0200
Subject: [Borgbackup] Does anyone backup VMs using private Linux
 VM-bridges with SSHFS?
In-Reply-To: <TYAPR01MB631321D39C998C2FAB91B2C8E11A9@TYAPR01MB6313.jpnprd01.prod.outlook.com>
References: <131618375.20210707090016@am-soft.de>
 <TYAPR01MB631321D39C998C2FAB91B2C8E11A9@TYAPR01MB6313.jpnprd01.prod.outlook.com>
Message-ID: <542699103.20210707115701@am-soft.de>

Guten Tag Heiko Helmle,
am Mittwoch, 7. Juli 2021 um 10:34 schrieben Sie:

> your biggest challenge is probably the large amount of data to
> parse by borg on every backup (basic proxmox backups are
> always-full), and the fact that borg is single-threaded.[...]

Seems like I wasn't clear: I wasn't talking about Proxmox-internal
backup, but really using Borg exclusively to backup VM contents. Those
VM are run by Proxmox and I would like to exec Borg itself on the
Proxmox host as well.

This shouldn't be too different from what I have done in the past:
Synology NAS + RSYNC logging into VMs using SSH and retrieving
individual files. That was single threaded one VM after another as
well and only needed a couple of minutes per VM most of the time.

The difference is that with that setup, there's a RSYNC process
running within the VMs. For various reasons I would like to make the
VM contents available to one Borg instance on Proxmox using SSHFS and
that increases traffic in theory.

Though, with some private bridge for all VMs for that backup purpose
only and not bridged to a physical NIC I was wondering if there'se any
relevant bottleneck already. I e.g. don't know what the max bandwith
and latency of such a bridge is. I would expect it's limited by CPU
only in the end, how fast the OS can build ethernet packets in RAM and
stuff.

> [...]I'd seriously look into proxmox
> backup server though, as it allows proxmox to do true incremental
> backups and greatly decreases the amount of data that gets moved around.

Thanks for the suggestion, have that in mind already, but no distinct
host to run it yet. Therefore I would like to keep my PULL-approach
for the time being, which doesn't sound too bad to me.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From lazyvirus at gmx.com  Wed Jul  7 09:59:15 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Wed, 7 Jul 2021 15:59:15 +0200
Subject: [Borgbackup] What are host and user name in archives used for?
In-Reply-To: <1647641287.20210707095440@am-soft.de>
References: <1647641287.20210707095440@am-soft.de>
Message-ID: <20210707155915.218102c4@msi.defcon1.lan>

On Wed, 7 Jul 2021 09:54:40 +0200
Thorsten Sch?ning<tschoening at am-soft.de> wrote:

> Hi all,

Hi alone,
 
> from my understanding of Borg, in Borg itself I only have two levels
> to organize my backups: The repo itself and multiple archives within.
> Sharing a repo increases the chance to deduplicate data amongst all
> contained archives, while each archive is something self contained,
> like a host or a backup purpose or alike.
[?]

Why don't you ie:

* standardize DIR names you want to exclude from backups on all machines
  (same name & case) + write a small exclusion file accordingly,

* mount all machines root to your borg server using sshfs, as of:
  /SSHFS/+---srv_borg		(this machine)
	 +---srv01
         +---srv02
         +--- ?
         +---proxmox01_on_srv02	(providing img dir is excluded
         |			 from srv02 backup)
         +---proxmox94_on_srv02 ( d? )
         +--- ?

* using only one borg repo for all machines, on the borg srv:
	borg create blablabla /BORG_REPO4ALL


However, if this way of doing things can save you some room 'coz of
dedup, it has a (big) flaw : if you have any problem on the repo that
need a repair, all of your backups can be badly impacted.


As my LAN is trusted and backups are only executed during the night, I've
chosen another solution :

* all clients mount their borg repo through NFS (which, for this
  direction, doesn't cause any inode problem)

* each client launches it's own script (exclusions have been
  standardized) and save it's backup to the NFS share

Using NFS avoids a CPU bottleneck that could be reached coz' of sshfs
crypto and scripts are launched under the root user id.

Some could argue that if a machine is compromised, the repo can be
savagely encrypted, but as everything's working under root, if "they"
compromise the root's account on a machine, it also means they probably
can do it on each and every other machine.

The advantage is, as each client is running it's own borg process, the
borg server is finally just a NAS, allowing multiple machines to backup
at once, providing rust is fast enough to absorb all flows at the same
time.

I lose room by opposition to a global dedup - although zlib(6)
compresses better than lz4 and is fast enough coz' it runs on each
client - but the hole shebang is fast and if there's any problem on a
repo, it only touches it's machine.

Just in case, NFS server is re-enabled each night by a cron, another
one disable it 2H later.

Jean-Yves

From tschoening at am-soft.de  Wed Jul  7 10:36:03 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Wed, 7 Jul 2021 16:36:03 +0200
Subject: [Borgbackup] What are host and user name in archives used for?
In-Reply-To: <20210707155915.218102c4@msi.defcon1.lan>
References: <1647641287.20210707095440@am-soft.de>
 <20210707155915.218102c4@msi.defcon1.lan>
Message-ID: <1789518252.20210707163603@am-soft.de>

Guten Tag Bzzzz,
am Mittwoch, 7. Juli 2021 um 15:59 schrieben Sie:

> However, if this way of doing things can save you some room 'coz of
> dedup, it has a (big) flaw : if you have any problem on the repo that
> need a repair, all of your backups can be badly impacted.

That's exactly one of the reasons I want to use multiple different
repos for multiple different, but similar content types.

I most likely backup VM images themself, their contents and some
database dumps and in my opinion it's somewhat unlikely that all of
those different content types share some blocks. So I wouldn't benefit
putting everything into one repo compared to use one repo for the
different content types. Files within the VM are VERY similar in my
case, while used file system etc. isn't, but influences the VM-image a
lot.

Besides that, archives themself seem to have limitations on the number
of files etc., so I don't want to make them unnecessary large right
from the start.

Don't get me too wrong: I can most likely live with two levels only,
didn't have more in the past as well. Just want to make sure to not
have misunderstood things and/or missed addiiutonal filtering
capatibilties.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Fri Jul  9 02:30:44 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Fri, 9 Jul 2021 08:30:44 +0200
Subject: [Borgbackup] Is ~100 MiB of new data OK for unchanged large image
 files?
Message-ID: <1489475951.20210709083044@am-soft.de>

Hi all,

I've started to test with Borg using BorMatic yesterday to see how
well it works with QCOW2 images I have. For that test, I backed up a
directory containing only one of those images with ~7 GiB of used
space and some small text file.

In the end, Borg behaves pretty much ike expected and only stores ~7
GiB in the repo. In case of repeated backup calls without any changes
to the image, the image gets rechunked, but not too much data added to
the repo. This is exactly how documented in the FAQ already:

https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me

Though, Borg does create some new chunks of data which are pretty much
~100 MiB each for each execution of Borg without any changes. It's not
that much of course, but from reading the FAQ I would have expected it
to be even less:

> This does not affect deduplication, the file will be chunked, but as
> the chunks will often be the same and already stored in the repo
> (except in the above mentioned rare condition), it will just re-use
> them as usual and not store new data chunks.

I've additionally read that chunks are created for things like file
names etc., buit in my case I only have on dir hierarchy and two files
to be backed up.

So, are ~100 MiB somewhat expected in that scenario of really very few
files with no changes?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tw at waldmann-edv.de  Fri Jul  9 05:53:18 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 9 Jul 2021 11:53:18 +0200
Subject: [Borgbackup] Is ~100 MiB of new data OK for unchanged large
 image files?
In-Reply-To: <1489475951.20210709083044@am-soft.de>
References: <1489475951.20210709083044@am-soft.de>
Message-ID: <e1ba1daa-fd40-e405-a336-7a88567cca22@waldmann-edv.de>

> In case of repeated backup calls without any changes
> to the image, the image gets rechunked, but not too much data added to
> the repo. This is exactly how documented in the FAQ already:
> 
> https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me

Yeah and especially the following entry might be relevant if it rechunks 
the qcow2 file without the qcow2 being modified:

https://borgbackup.readthedocs.io/en/stable/faq.html#i-am-seeing-a-added-status-for-an-unchanged-file

> Though, Borg does create some new chunks of data which are pretty much
> ~100 MiB each for each execution of Borg without any changes.

Sounds a bit much for no-change (would expect < 1MB) - can you open a 
github issue for this and give all info + script to reproduce it?

Did you start the VM that uses this qcow2 image in between the borg 
create runs?

Cheers, Thomas

-- 
GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Fri Jul  9 06:07:51 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 9 Jul 2021 12:07:51 +0200
Subject: [Borgbackup] What are host and user name in archives used for?
In-Reply-To: <1647641287.20210707095440@am-soft.de>
References: <1647641287.20210707095440@am-soft.de>
Message-ID: <ca658b39-9f93-b25d-2478-c556ac4a6d0f@waldmann-edv.de>

> from my understanding of Borg, in Borg itself I only have two levels
> to organize my backups: The repo itself and multiple archives within.

Yup. Just wanted to add that "multiple archives within" has at least 2 
sub cases:

1. multiple historical backups of same data set at different points in 
time, forming a sequence of archives that belong together

2. multiple backups of different data sets (from one machine, e.g. 
system vs. homes, from multiple machines, etc.) - each of them usually 
forming its own historical sequence.

This is all mapped into the archive name, usually like
{hostname}-{datasetname}-{timestamp} and has to be considered for 
--prefix option, esp. for borg prune.

> It's not clear to me currently if it's really only about a naming
> convention

Yes, currently there is no additional intelligence. Just archive names 
and helpers like placeholders to create nice names and stuff like 
--prefix to limit to a subset of all names.

> Additionally there's the env variable "BORG_HOST_ID" mentioning some
> automatically done calculations.

If you have different hostnames, I guess you do not need to use that.
It's just to have some unique ID, if you otherwise don't.

> In the end, it's about e.g. pruning old archives, which provides the
> argument "--prefix". From my understandig this really only applies to
> the name of the archive,

Yes.

> so I need to encode things like different
> backed up hosts, purposes etc. really only there or afterwards I won't
> be able to distinguish things when pruning anymore. E.g. when only
> using generic names with timestamps, I wouldn't be able to prune
> VM-contents more often than other backup purposes.

Yes, that is exactly why you need --prefix and a sane naming convention.

> Or are there mechanisms that e.g. additionally filter by fields
> "hostname", "username" etc. maintained by the archive itself? Maybe
> only some commands do so and prune does not?

IIRC we do not really use that metadata except keeping and displaying it.


-- 
GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Fri Jul  9 06:15:35 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 9 Jul 2021 12:15:35 +0200
Subject: [Borgbackup] How to store keyfiles outside repo AND home
 directory?
In-Reply-To: <937660494.20210706213741@am-soft.de>
References: <937660494.20210706213741@am-soft.de>
Message-ID: <1c1dd473-b757-fa77-6066-40fb67789754@waldmann-edv.de>

> So, is there some config I've missed to provide where to store the key
> files or do I need to work with symlinks/mounts/... to move them out
> of their default location?

See docs for BORG_KEYS_DIR.


-- 
GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tw at waldmann-edv.de  Fri Jul  9 06:27:50 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 9 Jul 2021 12:27:50 +0200
Subject: [Borgbackup] PULL+SSHFS: Avoid CHROOT with --numeric-owner?
In-Reply-To: <774305292.20210706213558@am-soft.de>
References: <774305292.20210706213558@am-soft.de>
Message-ID: <47118563-0c36-df26-f618-a3c2ae8e9fe6@waldmann-edv.de>

> So, is the CHROOT-stuff really only needed for that mapping

IIRC yes. Getting the mapping right (having correct names) was the idea.

This affects user/group name of owner/group from stat() and also ACLs.

Meanwhile we found that this attempt might be futile in some setups that 
do not exclusively use passwd/group files, but also other sources of 
name-id mappings (e.g. ldap).

> serving an additional purpose I simply didn't understand yet? In the
> first case, can CHROOT be skipped entirely by simply creating backups
> using "borg create --numeric-owner [...]"? Things sound easier to
> setup when not needing to care about CHROOT.

Yeah, esp. considering the recently found security implications...

Can you make a PR to change / clarify the docs?

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From tschoening at am-soft.de  Fri Jul  9 12:59:46 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Fri, 9 Jul 2021 18:59:46 +0200
Subject: [Borgbackup] "type fuse" instead of "type fuse.borgfs" in output of
 "mount"
Message-ID: <1277711811.20210709185946@am-soft.de>

Hi all,

to make a long story short, I recognized the following different
output of "mount" during some tests:

> u269594 at u269594.your-storagebox.de: on /mnt/bak_restore/sbox type fuse.sshfs ([...])
> borgfs on /mnt/bak_restore/borgfs/pve type fuse ([...])

Is it correct that for borgfs the type is only "fuse" instead of
"fuse.borgfs"?

The latter would be inline with SSHFS and the example for /etc/fstab
in the docs:

> /path/to/repo /mnt/point fuse.borgfs defaults,noauto 0 0

https://borgbackup.readthedocs.io/en/stable/usage/borgfs.html?highlight=borgfs#description

I'm scripting a bit with mounting multiple repos at once and how to
easily unmount those without knowing their paths. Found the following,
which unmounts my above example of borgfs while keeping SSHFS. So it's
pretty much exactly what I want, though "fuse" alone seems unnecessary
generic to me.

> umount --all --types fuse

vs.

> umount --all --types fuse.borgfs # Doesn't work!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Fri Jul  9 13:04:33 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Fri, 9 Jul 2021 19:04:33 +0200
Subject: [Borgbackup] borgfs vs. sudo: Can't answer relocation confirmation
Message-ID: <8910075015.20210709190433@am-soft.de>

Hi all,

I have a script mounting some repos using borgfs under some special
backup/restore user using sudo. That seems to work fine at first,
borgfs successfully asks for a password, I enter one and afterwards
the warning about a relocated repo occurs and aks me if I want to
continue or not.

And here's the problem: Whatever I do, borgfs(?) simply doesn't wait
for me providing the answer. It instantly aborts and the mount fails.
What I needed to do is setting an environment variable answering that
question and mountg succeeds.

> sudo -u $USER borgfs FROM $TO

vs.

> sudo -u $USER BORG_RELOCATED_REPO_ACCESS_IS_OK=y borgfs FROM $TO

I guess it has something to do with sudo, but as I'm able to provide a
password, I wonder what the difference is to the confirmation
question.

Any idea? Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From lazyvirus at gmx.com  Fri Jul  9 13:33:24 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Fri, 9 Jul 2021 19:33:24 +0200
Subject: [Borgbackup] borgfs vs. sudo: Can't answer relocation
 confirmation
In-Reply-To: <8910075015.20210709190433@am-soft.de>
References: <8910075015.20210709190433@am-soft.de>
Message-ID: <20210709193324.2044ad34@msi.defcon1.lan>

On Fri, 9 Jul 2021 19:04:33 +0200
Thorsten Sch?ning<tschoening at am-soft.de> wrote:

> Hi all,

Hi alone,

> I have a script mounting some repos using borgfs under some special
> backup/restore user using sudo. That seems to work fine at first,
> borgfs successfully asks for a password, I enter one and afterwards
> the warning about a relocated repo occurs and aks me if I want to
> continue or not.

This is because this user have had worked onto other DIRs as repos.
Just acknowledge and it'll rewrite the old one with the new one and
cease to complain.

> And here's the problem: Whatever I do, borgfs(?) simply doesn't wait
> for me providing the answer. It instantly aborts and the mount fails.
> What I needed to do is setting an environment variable answering that
> question and mountg succeeds.

You must supply the p/w in your script (in bash) :

?
# Test if my bootyfull borg server is up|down
(exec ping -c2 my.borg.server.that.is.a.server.mydomain)
[ $? -ne 0 ]; then
    clear
    echo "\n\nBORG server is DOWN - aborting backup.\n\n"
    exit 1
fi
?

Watch it: never ever put something else between the ping lines and the
test block, otherwise it will kill the ping command result.

> I guess it has something to do with sudo, but as I'm able to provide a
> password, I wonder what the difference is to the confirmation
> question.

Why not a cron launch, that allows for any user to run a script.
 
> Any idea? Thanks!
> 
>  Mit freunden Gr??lichen

Jean-Yves

From lazyvirus at gmx.com  Fri Jul  9 15:22:27 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Fri, 9 Jul 2021 21:22:27 +0200
Subject: [Borgbackup] borgfs vs. sudo: Can't answer relocation
 confirmation
In-Reply-To: <8910075015.20210709190433@am-soft.de>
References: <8910075015.20210709190433@am-soft.de>
Message-ID: <20210709212227.37e8e061@msi.defcon1.lan>

On Fri, 9 Jul 2021 19:04:33 +0200
Thorsten Sch?ning<tschoening at am-soft.de> wrote:

OOPS, wrong code, my bad !

> 
> And here's the problem: Whatever I do, borgfs(?) simply doesn't wait
> for me providing the answer. It instantly aborts and the mount fails.
> What I needed to do is setting an environment variable answering that
> question and mountg succeeds.

# exporting repo p/w
export BORG_PASSPHRASE="password"
borg mount ?

but I'm not sure one instance of borg can cope with different passwords
without only keeping the last one.

Jean-Yves

From lazyvirus at gmx.com  Fri Jul  9 15:29:19 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Fri, 9 Jul 2021 21:29:19 +0200
Subject: [Borgbackup] "type fuse" instead of "type fuse.borgfs" in
 output of "mount"
In-Reply-To: <1277711811.20210709185946@am-soft.de>
References: <1277711811.20210709185946@am-soft.de>
Message-ID: <20210709212919.72c0394f@msi.defcon1.lan>

On Fri, 9 Jul 2021 18:59:46 +0200
Thorsten Sch?ning<tschoening at am-soft.de> wrote:

> The latter would be inline with SSHFS and the example for /etc/fstab
> in the docs:
> 
> > /path/to/repo /mnt/point fuse.borgfs defaults,noauto 0 0

Try with: fuse.sshfs
instead.
 
Jean-Yves

From tschoening at am-soft.de  Sat Jul 10 09:04:45 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Sat, 10 Jul 2021 15:04:45 +0200
Subject: [Borgbackup] What's the least amount of data necessary to access
 repos?
Message-ID: <1338223783.20210710150445@am-soft.de>

Hi everyone,

I've currently setup some borg repos using the "keyfile" encryption
method. From my understanding this forces me to maintain the used
password during "init" AND additionally the generated keyfiles to be
able to access the repo.

I've chosen this because from my understanding it's the most secure
setup one can have. OTOH, in case of switched backup hosts or alike,
this setup is a bit more difficult to restore: While borg asks for
passwords in the easiest case, the key files need to be made available
at some special location, with the correct name etc. Not that of a big
deal, though, but I'm thinking about if I really want this or not.

Am I correct that with a setup like "repokey" the only thing I would
need to keep is the password? Nothing else from the user profile
executing backups and created by borg automatically?

Or am I missing something and some additional data would always be
required as well? In that case "keyfile" might not make much
difference anymore.

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tw at waldmann-edv.de  Sat Jul 10 10:10:46 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sat, 10 Jul 2021 16:10:46 +0200
Subject: [Borgbackup] What's the least amount of data necessary to
 access repos?
In-Reply-To: <1338223783.20210710150445@am-soft.de>
References: <1338223783.20210710150445@am-soft.de>
Message-ID: <a3b22e83-440b-5597-f531-7068f41be5c9@waldmann-edv.de>


> I've currently setup some borg repos using the "keyfile" encryption
> method. From my understanding this forces me to maintain the used
> password during "init" AND additionally the generated keyfiles to be
> able to access the repo.

To decrypt data from an encrypted borg repo, you always need the borg 
AES key AND (because that key is protected by another layer of 
encryption) the passphrase for that key.

The only difference between repokey and keyfile is where the key is stored:

In the repo dir (repo/config file) vs. in the home dir.

You can easily lose the home dir, e.g. if the machine is stolen or the 
disk dies.

But you could also lose they key inside the repo dir (e.g. due to a hw 
issue, fs issue, sw bug, ...) and it would be painful to still have 
repodir/data but no key to decrypt.

So, you want to have a key backup for both cases and also not lose the 
passphrase.

borg key export --qr-html repo1 key-repo1.html

and then open key-repo1.html with a browser and print the page.

the passphrase is intentionally NOT included in the printed data.


From gasmit at xs4all.nl  Mon Jul 12 08:44:31 2021
From: gasmit at xs4all.nl (gasmit at xs4all.nl)
Date: Mon, 12 Jul 2021 14:44:31 +0200
Subject: [Borgbackup] NLUUG vEvent :: 2021-07-15 20:30 CET :: Thomas
 Waldmann: "Borg Backup 2/2: You will be assimilated"
Message-ID: <f6601ba3-a897-c1b0-3cee-1f7d79f49de3@ophogepoten.nl>

s/Conferences/vEvents/g


Hello,


Due to circumstances, the NLUUG organized some virtual events.

In the next one, Thomas speaks English,
so you might like to attend his presentation, next Thursday.


Abstract:


BorgBackup (short: Borg) is a deduplicating backup program. Optionally,
it supports compression and authenticated encryption.

This vEvent will be about BorgBackup's algorithms, data structures,
administration and optimization, ...

Also, I'll cover some FAQs that usually come up when practically using
borg.

Note:
After the introductory talk in the last meetup, I'll go a bit deeper
this time.

If you did not use borg yet and if you didn't attend the last talk either,
it is recommended that you have a look at borgbackup.org and maybe do
some own experiments as a preparation.


About the speaker:

*1966

My interest in IT started with programmable calculators (Ti, Casio Basic),
then had C64 + tape drive (Basic, Assembler, Pascal), Atari ST (Logo,
Assembler, C, Prolog, ...), Transputer (Occam), i386 (C under DOS+Win31,
what a pain), then OS/2 (finally: a real OS and 32bits, C/C++, REXX),
then to Linux (C/C++, Python, shell, ...).

Trying to avoid Windows. :-)

I studied CS ("Diplom Informatiker"), focus hardware and computational
linguistics at University of Stuttgart and am running my own small IT
company since then with my brother.

My focus nowadays is on Linux, Servers, Hardware, developing SW, usually
in Python and for FOSS projects, like e.g.: BorgBackup, MoinMoin Wiki,
bepasty, nsupdate-dot-info, xstatic, ...


When: Thursday July 15. 2021
Agenda:
19:30 Doors open @ https://meet.nluug.nl/b/nlu-wpg-sgj
20:30 Opening
20:35 Presentation
21:20 Questions
22:00 Closing

Here you can test your equipment:
https://nl.webcamtests.com/
https://nl.mictests.com/
You do not need those to participate.

A recording of the vEvent will be made available, see nluug.nl/vEvents


CU next Thursday!


With kind regards,
The people of the NLUUG


-- 
Gerrit A. Smit
NLUUG program committee


The NLUUG unites professonal users of
Opens Systems and Open Standards in the Netherlands

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210712/3caefd2a/attachment.html>

From tw at waldmann-edv.de  Mon Jul 12 17:08:41 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 12 Jul 2021 23:08:41 +0200
Subject: [Borgbackup] borgbackup 1.1.17 released
Message-ID: <12bc42c9-6ba1-beda-e1f7-e69045cc7310@waldmann-edv.de>

Just released borgbackup 1.1.17 with some fixes and improvements, see 
there for details:

https://github.com/borgbackup/borg/releases/tag/1.1.17


-- 
GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393

From gasmit at xs4all.nl  Tue Jul 13 02:40:32 2021
From: gasmit at xs4all.nl (gasmit at xs4all.nl)
Date: Tue, 13 Jul 2021 08:40:32 +0200
Subject: [Borgbackup] NLUUG vEvent :: 2021-07-15 20:30 CET :: Thomas
 Waldmann: "Borg Backup 2/2: You will be assimilated" :: Timezone is CEST
In-Reply-To: <f6601ba3-a897-c1b0-3cee-1f7d79f49de3@ophogepoten.nl>
References: <f6601ba3-a897-c1b0-3cee-1f7d79f49de3@ophogepoten.nl>
Message-ID: <5f9623e7-7046-ec4d-c34c-f1ac6bfcc1c7@xs4all.nl>

Good morning (?),


Times are correct for CEST.

> When: Thursday July 15. 2021
> Agenda:
> 19:30 Doors open @ https://meet.nluug.nl/b/nlu-wpg-sgj
> 20:30 Opening
> 20:35 Presentation
> 21:20 Questions
> 22:00 Closing


-- 
Gerrit A. Smit
NLUUG program committee

De NLUUG verenigt professionele gebruikers van
Open Systemen en Open Standaarden in Nederland

The NLUUG unites professional users of
Opens Systems and Open Standards in the Netherlands

P.S.
Thank you, Jonas.

From cherio at gmail.com  Tue Jul 13 15:26:04 2021
From: cherio at gmail.com (Cherio)
Date: Tue, 13 Jul 2021 15:26:04 -0400
Subject: [Borgbackup] Remote backups via ssh - OpenSSL version mismatch
In-Reply-To: <CAKHqFkJEhKvNDUFAwnxHEG2_U47bxLm0ND+G11=8T-_kdCxRHw@mail.gmail.com>
References: <CAKHqFkKOFUHoATqW8HXzftdtoci8OxFbaaw79H-43i2GYEpUzg@mail.gmail.com>
 <bf7ded9f-fe8b-937c-bb51-18bcc3ecb0ef@waldmann-edv.de>
 <CAKHqFkJ7fV7FCDAe7XN_wQ9u3PP8S41VkO2qzC+ko+-a4AhcYw@mail.gmail.com>
 <a1bbfdbe-3623-ad01-467d-dbe7392f51f7@waldmann-edv.de>
 <485b1a15-b625-1d72-0802-1f0bb1a8a79f@waldmann-edv.de>
 <2c62cf95-4ebc-c78d-922d-9acc6aa5def4@waldmann-edv.de>
 <CAKHqFkJEhKvNDUFAwnxHEG2_U47bxLm0ND+G11=8T-_kdCxRHw@mail.gmail.com>
Message-ID: <CAKHqFkLQs3xon=7d1NJPZzi5kGT9MznMYYndaeFiofJavCeOSA@mail.gmail.com>

This issue is no longer present in version 1.1.17.
Backups via SSH with a directory extracted (borg-linux64.tgz) version run
without issues again.

Thank you Thomas and those who fixed it.

On Sun, Jul 4, 2021 at 3:28 PM Cherio <cherio at gmail.com> wrote:

> Your suspicion is correct!!!!!!!
>
> The single file 1.1.16 linux executable works. The directory
> extracted borg-linux64.tgz doesn't.
>
> On Sun, Jul 4, 2021 at 5:23 AM Thomas Waldmann <tw at waldmann-edv.de> wrote:
>
>> >> Can you try the single-file bundle also for 1.1.14 and 1.1.16?
>> >
>> > Ignore that, I've read in your first post that you have seen the bug
>> > also in the single-file bundle.
>>
>> Hmm, i did some experiments and think that:
>> - 1.1.16 single-dir pyinstaller bundle has that bug
>> - 1.1.16 single-file pyinstaller bundle should work, no bug
>>
>> @cherio can you try again please and confirm?
>>
>> > https://github.com/borgbackup/borg/issues/5897
>>
>> If you have a github account, we could also continue there.
>>
>> --
>>
>> GPG ID: 9F88FB52FAF7B393
>> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393
>> _______________________________________________
>> Borgbackup mailing list
>> Borgbackup at python.org
>> https://mail.python.org/mailman/listinfo/borgbackup
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210713/5997c22c/attachment.html>

From lazyvirus at gmx.com  Wed Jul 14 21:40:43 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Thu, 15 Jul 2021 03:40:43 +0200
Subject: [Borgbackup] Lock question
Message-ID: <20210715034043.70853315@msi.defcon1.lan>

Hi compressed and deduplicated people,

I wonder why "borg list" cannot work when a backup is running,
complaining it can't create/acquire the lock, as listing backups just
read data with no write (AFAIK) ?

Jean-Yves

From tschoening at am-soft.de  Thu Jul 22 13:31:47 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Thu, 22 Jul 2021 19:31:47 +0200
Subject: [Borgbackup] Is there a way to update an existing archive?
Message-ID: <798530473.20210722193147@am-soft.de>

Hi everyone,

I'm trying a file system backup of postgres, which roughly consists of
the following steps:

> psql -c "select pg_start_backup('hot_backup');"
> borg [...]
> psql -c "select pg_stop_backup();"
> ??? -> Last WAL file needed!

https://www.postgresql.org/docs/11/continuous-archiving.html#BACKUP-TIPS

The important thing is that I need to add some special files into the
archive after pg_stop_backup has been executed, which only runs after
borg has finished already. Otherwise the archive is inconsistent. Of
course there are workarounds like simply creating an additional
archive afterwards, but I would prefer having one consinstent archive
of course.

So, is there any support for such use case? Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From tschoening at am-soft.de  Thu Jul 22 15:04:36 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Thu, 22 Jul 2021 21:04:36 +0200
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <798530473.20210722193147@am-soft.de>
References: <798530473.20210722193147@am-soft.de>
Message-ID: <684043935.20210722210436@am-soft.de>

Guten Tag Thorsten Sch?ning,
am Donnerstag, 22. Juli 2021 um 19:31 schrieben Sie:

>> psql -c "select pg_start_backup('hot_backup');"
>> borg [...]
>> psql -c "select pg_stop_backup();"
>> ??? -> Last WAL file needed!

> So, is there any support for such use case? Thanks!

In theory, checkpoints read pretty much like what I would need: The
first backup up to "pg_stop_backup" might be considered as the first
checkpoint. Backup is interrupted, "pg_stop_backup" executed and "borg
create" executed again to create the overall backup including the last
WAL. The second execution would work based on the data associated with
the former checkpoint, but create the overall correct archive in the
end.

Is there any way to make use of checkpoints like this? The docs read
like they are maintained automatically mostly and created, pruned etc.
as needed. I don't see any commands to abort with a checkpoint, one
needs to kille a process and stuff like that most likely?

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From lazyvirus at gmx.com  Thu Jul 22 15:49:41 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Thu, 22 Jul 2021 21:49:41 +0200
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <684043935.20210722210436@am-soft.de>
References: <798530473.20210722193147@am-soft.de>
 <684043935.20210722210436@am-soft.de>
Message-ID: <20210722214941.75e22c43@msi.defcon1.lan>

On Thu, 22 Jul 2021 21:04:36 +0200
Thorsten Sch?ning<tschoening at am-soft.de> wrote:

> Guten Tag Thorsten Sch?ning,
> am Donnerstag, 22. Juli 2021 um 19:31 schrieben Sie:
> 
> >> psql -c "select pg_start_backup('hot_backup');"
> >> borg [...]
> >> psql -c "select pg_stop_backup();"
> >> ??? -> Last WAL file needed!
> 
> > So, is there any support for such use case? Thanks!
> 
> In theory, checkpoints read pretty much like what I would need: The
> first backup up to "pg_stop_backup" might be considered as the first
> checkpoint. Backup is interrupted, "pg_stop_backup" executed and "borg
> create" executed again to create the overall backup including the last
> WAL. The second execution would work based on the data associated with
> the former checkpoint, but create the overall correct archive in the
> end.
> 
> Is there any way to make use of checkpoints like this? The docs read
> like they are maintained automatically mostly and created, pruned etc.
> as needed. I don't see any commands to abort with a checkpoint, one
> needs to kille a process and stuff like that most likely?

I'd say lower the default "--checkpoint-interval 1800", let's say down to
600 (seconds) and add timers into your script to make sure you cut BB
with a security margin ((600s ? X) + 30~60s), do what you have to do and
re-launch BB.

In any case, carefully check this is working as expected.

Jean-Yves

From tw at waldmann-edv.de  Thu Jul 22 19:45:43 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 23 Jul 2021 01:45:43 +0200
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <798530473.20210722193147@am-soft.de>
References: <798530473.20210722193147@am-soft.de>
Message-ID: <6b081bb7-ed4e-ac58-daad-50bae06ddb08@waldmann-edv.de>

 > So, is there any support for such use case? Thanks!

No, you can't add anything to an archive afterwards.



From billk at iinet.net.au  Thu Jul 22 20:06:10 2021
From: billk at iinet.net.au (William Kenworthy)
Date: Fri, 23 Jul 2021 08:06:10 +0800
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <6b081bb7-ed4e-ac58-daad-50bae06ddb08@waldmann-edv.de>
References: <798530473.20210722193147@am-soft.de>
 <6b081bb7-ed4e-ac58-daad-50bae06ddb08@waldmann-edv.de>
Message-ID: <19494892-b783-9723-c361-aecb8699d14c@iinet.net.au>

Create the tar file as per the hot backup steps then just stuff that
into borg when complete and delete it afterwards? Sidesteps the issue.

BillK


On 23/7/21 7:45 am, Thomas Waldmann wrote:
> > So, is there any support for such use case? Thanks!
>
> No, you can't add anything to an archive afterwards.
>
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup

From tschoening at am-soft.de  Fri Jul 23 03:21:19 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Fri, 23 Jul 2021 09:21:19 +0200
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <19494892-b783-9723-c361-aecb8699d14c@iinet.net.au>
References: <798530473.20210722193147@am-soft.de>
 <6b081bb7-ed4e-ac58-daad-50bae06ddb08@waldmann-edv.de>
 <19494892-b783-9723-c361-aecb8699d14c@iinet.net.au>
Message-ID: <1334392604.20210723092119@am-soft.de>

Guten Tag William Kenworthy,
am Freitag, 23. Juli 2021 um 02:06 schrieben Sie:

> Create the tar file as per the hot backup steps then just stuff that
> into borg when complete and delete it afterwards? Sidesteps the issue.

In my case that would require hundreds of GiB temporary data somewhere
and leave me with one large file in the archive. While backing up the
entire data directory as-is is possible without additional temporary
data and in theory, using BorgFS one might even be able to run
Postgres read-only directly on the archive. I guess de-duplication
and compression works a bit better with individual files as well,
especially as some files might not even change at all when read
directly by Borg.

At some point I plan to use BTRFS or ZFS for underlying storage of
Postgres anyway, which should allow me to run all necessary
Postgres-command before executing Borg.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From billk at iinet.net.au  Fri Jul 23 04:11:11 2021
From: billk at iinet.net.au (William Kenworthy)
Date: Fri, 23 Jul 2021 16:11:11 +0800
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <1334392604.20210723092119@am-soft.de>
References: <798530473.20210722193147@am-soft.de>
 <6b081bb7-ed4e-ac58-daad-50bae06ddb08@waldmann-edv.de>
 <19494892-b783-9723-c361-aecb8699d14c@iinet.net.au>
 <1334392604.20210723092119@am-soft.de>
Message-ID: <daa1cee4-db41-7db8-6967-aa136db6d9da@iinet.net.au>

On 23/7/21 3:21 pm, Thorsten Sch?ning wrote:
> Guten Tag William Kenworthy,
> am Freitag, 23. Juli 2021 um 02:06 schrieben Sie:
>
>> Create the tar file as per the hot backup steps then just stuff that
>> into borg when complete and delete it afterwards? Sidesteps the issue.
> In my case that would require hundreds of GiB temporary data somewhere
> and leave me with one large file in the archive. While backing up the
> entire data directory as-is is possible without additional temporary
> data and in theory, using BorgFS one might even be able to run
> Postgres read-only directly on the archive. I guess de-duplication
> and compression works a bit better with individual files as well,
> especially as some files might not even change at all when read
> directly by Borg.
>
> At some point I plan to use BTRFS or ZFS for underlying storage of
> Postgres anyway, which should allow me to run all necessary
> Postgres-command before executing Borg.
>
> Mit freundlichen Gr??en
>
> Thorsten Sch?ning
>
I use snapshots to fix a copy of the data for that (e.g., you could
snapshot the whole postgres data and configuration areas - uses no
actual space on btrfs and moosefs), then stuff the snapshot into a borg
repo before deleting it.? See
"https://dba.stackexchange.com/questions/44260/postgresql-freeze-unfreeze-command-equivalents"
for comments on taking an atomic snapshot.? Snapshots are working with
multiple terabytes of data for me.

I have investigated but not tried past testing the concept is the option
of piping the data (e.g., dump tarfile plus others) into borg create via
a pipe (search the docs about "borg create" for the word "pipe").
Similarly, it should also be possible to create a named pipe connected
to a borg create instance and just keep dropping files into it?? For
ideas on different ways to approach this,? look into how dd, tar and
bzip2 can be used to clone an OS across a network using pipes without
affecting file names, permissions etc.

Something to keep in mind - "borg extract" on a whole dataset in a repo
is very fast while borgFS is much slower but its easier to locate and
extract single files/smaller amounts of data.

BillK



From tschoening at am-soft.de  Fri Jul 23 04:56:04 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Fri, 23 Jul 2021 10:56:04 +0200
Subject: [Borgbackup] Is there a way to update an existing archive?
In-Reply-To: <daa1cee4-db41-7db8-6967-aa136db6d9da@iinet.net.au>
References: <798530473.20210722193147@am-soft.de>
 <6b081bb7-ed4e-ac58-daad-50bae06ddb08@waldmann-edv.de>
 <19494892-b783-9723-c361-aecb8699d14c@iinet.net.au>
 <1334392604.20210723092119@am-soft.de>
 <daa1cee4-db41-7db8-6967-aa136db6d9da@iinet.net.au>
Message-ID: <407722200.20210723105604@am-soft.de>

Guten Tag William Kenworthy,
am Freitag, 23. Juli 2021 um 10:11 schrieben Sie:

> [...]See
> "https://dba.stackexchange.com/questions/44260/postgresql-freeze-unfreeze-command-equivalents"
> for comments on taking an atomic snapshot.? Snapshots are working with
> multiple terabytes of data for me.

Which of the approaches are you using actually? Crash consistency with
FS-level snapshot only or pg_start_backup+pg_stop_backup+copy+WAL?

> I have investigated but not tried past testing the concept is the option
> of piping the data (e.g., dump tarfile plus others) into borg create via
> a pipe (search the docs about "borg create" for the word
> "pipe").[...]

I'm using BorgMatic with its hooks, which are simply apps/scripts
executed before and after "borg create" and therefore create some
lifecycle which normally makes totally sense to e.g. create FS-level
snapshots and clean those up. Things like pipes are a bit more
difficult, because while those could be created BEFORE and used by
"borg create", AFTER wouldn't fire to provide the WALs created by
pg_stop_backup. Some kind of chicken&egg problem.

But nevermind, it's not too important anymore, I'm more interested in
your FS-level snapshots. I most likely won't run systems without those
anymore.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From oh at dom.de  Fri Jul 23 09:12:13 2021
From: oh at dom.de (Oliver Hoffmann)
Date: Fri, 23 Jul 2021 15:12:13 +0200
Subject: [Borgbackup] How to delete an archive without passphrase for key?
Message-ID: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>

Hi list,


I have some obsolete archives from old and long gone clients. I can't do
 borg delete /path/to/archive, because the passphrase for the key is of
course lost too.
What now? I just delete those folders with rm -rf /path/to/archive?

Cheers,

Oliver

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210723/3c3d3fab/attachment.sig>

From tw at waldmann-edv.de  Fri Jul 23 09:36:34 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 23 Jul 2021 15:36:34 +0200
Subject: [Borgbackup] How to delete an archive without passphrase for
 key?
In-Reply-To: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>
References: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>
Message-ID: <a832b55c-6b71-a81e-a4df-917cb4e3748f@waldmann-edv.de>

I suspect you mean borg REPOSITORIES (not archives)?

If so, see there: https://github.com/borgbackup/borg/issues/5941

On 23.07.21 15:12, Oliver Hoffmann wrote:
> I have some obsolete archives from old and long gone clients. I can't do
>   borg delete /path/to/archive, because the passphrase for the key is of
> course lost too.
> What now? I just delete those folders with rm -rf /path/to/archive?

From oh at dom.de  Fri Jul 23 09:50:36 2021
From: oh at dom.de (Oliver Hoffmann)
Date: Fri, 23 Jul 2021 15:50:36 +0200
Subject: [Borgbackup] How to delete an archive without passphrase for
 key?
In-Reply-To: <a832b55c-6b71-a81e-a4df-917cb4e3748f@waldmann-edv.de>
References: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>
 <a832b55c-6b71-a81e-a4df-917cb4e3748f@waldmann-edv.de>
Message-ID: <3ac2a983-cd3e-2b21-a4c1-96e47521e4f3@dom.de>

Hi Thomas,


thanks for the quick reply.
Yes, sorry for the confusion. Repo is what I meant.

Well, I can't find these folders:

~/.cache/borg/
~/.config/borg/security/

Not as root, not as any other user.

I got that ID for my repo but this is nowhere to find as well.

What am I missing?

Greetings,

Oliver


> I suspect you mean borg REPOSITORIES (not archives)?
> 
> If so, see there: https://github.com/borgbackup/borg/issues/5941
> 
> On 23.07.21 15:12, Oliver Hoffmann wrote:
>> I have some obsolete archives from old and long gone clients. I can't do
>> ? borg delete /path/to/archive, because the passphrase for the key is of
>> course lost too.
>> What now? I just delete those folders with rm -rf /path/to/archive?
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup


-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210723/e1eb4ba3/attachment-0001.sig>

From tw at waldmann-edv.de  Fri Jul 23 15:09:51 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Fri, 23 Jul 2021 21:09:51 +0200
Subject: [Borgbackup] How to delete an archive without passphrase for
 key?
In-Reply-To: <3ac2a983-cd3e-2b21-a4c1-96e47521e4f3@dom.de>
References: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>
 <a832b55c-6b71-a81e-a4df-917cb4e3748f@waldmann-edv.de>
 <3ac2a983-cd3e-2b21-a4c1-96e47521e4f3@dom.de>
Message-ID: <52b17528-bb1f-05d1-9da7-9cda61d5f182@waldmann-edv.de>


> Well, I can't find these folders:
>
> ~/.cache/borg/
> ~/.config/borg/security/
>
> Not as root, not as any other user.

It's their default location, but one can change this with environment 
variables - maybe check your backup script / shell startup files for 
BORG_* env variables.

Also note that all files / directories starting with a dot are hidden by 
most file managers / shell, so e.g. on the shell you need ls -la to see 
them.



From jdc at uwo.ca  Fri Jul 23 15:39:45 2021
From: jdc at uwo.ca (Dan Christensen)
Date: Fri, 23 Jul 2021 19:39:45 +0000
Subject: [Borgbackup] How to delete an archive without passphrase for
 key?
In-Reply-To: <3ac2a983-cd3e-2b21-a4c1-96e47521e4f3@dom.de> (Oliver Hoffmann's
 message of "Fri, 23 Jul 2021 15:50:36 +0200")
References: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>
 <a832b55c-6b71-a81e-a4df-917cb4e3748f@waldmann-edv.de>
 <3ac2a983-cd3e-2b21-a4c1-96e47521e4f3@dom.de>
Message-ID: <87sg04n7jz.fsf@uwo.ca>

On Jul 23, 2021, Oliver Hoffmann <oh at dom.de> wrote:

> Well, I can't find these folders:
>
> ~/.cache/borg/
> ~/.config/borg/security/
>
> Not as root, not as any other user.

Those folders would be on the clients, so if the clients are long gone
as you said, then you don't need to worry about them.

Dan

From oh at dom.de  Mon Jul 26 08:08:44 2021
From: oh at dom.de (Oliver Hoffmann)
Date: Mon, 26 Jul 2021 14:08:44 +0200
Subject: [Borgbackup] How to delete an archive without passphrase for
 key?
In-Reply-To: <87sg04n7jz.fsf@uwo.ca>
References: <81ecce59-83e7-b1aa-06a3-0106db38f31a@dom.de>
 <a832b55c-6b71-a81e-a4df-917cb4e3748f@waldmann-edv.de>
 <3ac2a983-cd3e-2b21-a4c1-96e47521e4f3@dom.de> <87sg04n7jz.fsf@uwo.ca>
Message-ID: <43151108-06c8-2233-f3df-a33d4742bd82@dom.de>

Ah, that's what I suspected.
All right then it's just rm -rf on the server.
Thank you!

Cheers,

Oliver

> On Jul 23, 2021, Oliver Hoffmann <oh at dom.de> wrote:
> 
>> Well, I can't find these folders:
>>
>> ~/.cache/borg/
>> ~/.config/borg/security/
>>
>> Not as root, not as any other user.
> 
> Those folders would be on the clients, so if the clients are long gone
> as you said, then you don't need to worry about them.
> 
> Dan
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
> 


-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210726/84144307/attachment.sig>

From mason at ftlcomputing.com  Mon Jul 26 14:10:53 2021
From: mason at ftlcomputing.com (Mason Schmitt)
Date: Mon, 26 Jul 2021 11:10:53 -0700
Subject: [Borgbackup] (no subject)
Message-ID: <CAM_s9ZnM1MR4DVvPyAef9b0+ik6zP677SWNyrU74po07jPv0Bg@mail.gmail.com>

Hello,

OS - CentOS 7.9.2009
Borg - 1.1.16 (from the EPEL repo)
Filesystem - XFS on LVM
VM images - raw and qcow2 images on the XFS filesystem
VM image sizes - OS images are between 15GB and 100GB.  Data images can be
upwards of 2TB.
Backup process - Freeze VM fileystems, create LVM snapshots, mount
snapshots, run Borg against the mount point, unmount, delete snapshots


I have Borg running on over a dozen systems, all at different locations.  I
use it to backup VM images to locally attached USB drives.  Overall Borg
has been very reliable.

However, I do have one site where I've been experiencing backup failures.
Each time it has happened, I have taken increasingly drastic measures to
get the backups working again.  I'll share what I've done to troubleshoot
the issue in the past and provide more details about the most recent
failure, which started last week.


May 3 - 1st occurrence
-----------------------------------
Saw this in the log:
<lots of python error messages>
borg.helpers.IntegrityError: Data integrity error: Segment entry checksum
mismatch [segment 11589, offset 267452375]

Ran:
borg check -p -v --repair /mnt/borg/borg-backups/backup.borg/

The backups seemed to be ok for about a week, but then when the weekly
`borg check` was run, it reported tons of errors.

I had seen a single badblock report for the USB drive, so I replaced the
USB backup drive and initialized a new borg repo and things seemed to be ok
for two weeks.

As an aside - I ran badblocks against the old USB drive and it didn't find
any issues with the drive.


June 14 - 2nd occurrence
----------------------------------------------
Backups failed and I again saw tons of errors in the log, following a run
of `borg check`.

Decided to take the nuclear option:
- Deleted /root/.cache/borg
- Deleted /root/.config/borg
- Wiped the USB backup drive and re-initialized the repo

The backups again started to work properly and the weekly `borg check` also
seemed to be happy.


July 18 - 3rd occurrence
---------------------------------------------
Backups failed and continued to fail each day with very similar error
messages to what I saw on May 3.  See below for details.


What should I do?
-------------------------------
Given all of my sites are very similar (very similar hardware, same OS,
same Borg version, same configuration, all backing up VM images) and given
that I'm not having issues at any other sites; I'm guessing my Borg and LVM
snapshot configuration is ok.  So, does anyone have insight into what might
be going on at this site and what steps I could take to troubleshoot it?



Sun Jul 18 22:24:26 PDT 2021 Starting backup
borgbackup version 1.1.16
Creating archive at
"/mnt/borg/borg-backups/backup.borg::2021-07-18-fqdn-25793-system"

.... several lines ommitted .....


M /mnt/snapbak/var/lib/libvirt/images/vm_slow_data/fs1-DATA.qcow2
Data integrity error: Segment entry checksum mismatch [segment 2438,
offset 9906907]
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4690, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line 4622, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
177, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
595, in do_create
    create_inner(archive, cache)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
555, in create_inner
    read_special=args.read_special, dry_run=dry_run, st=st)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
672, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
672, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
672, in _process
    read_special=read_special, dry_run=dry_run)
  [Previous line repeated 3 more times]
  File "/usr/lib64/python3.6/site-packages/borg/archiver.py", line
646, in _process
    status = archive.process_file(path, st, cache)
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line
1091, in process_file
    self.chunk_file(item, cache, self.stats,
backup_io_iter(self.chunker.chunkify(fd, fh)))
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line
1012, in chunk_file
    from_chunk, part_number = self.write_part_file(item, from_chunk,
part_number)
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 990,
in write_part_file
    self.write_checkpoint()
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 483,
in write_checkpoint
    self.save(self.checkpoint_name)
  File "/usr/lib64/python3.6/site-packages/borg/archive.py", line 530, in save
    self.repository.commit()
  File "/usr/lib64/python3.6/site-packages/borg/repository.py", line
475, in commit
    self.compact_segments()
  File "/usr/lib64/python3.6/site-packages/borg/repository.py", line
756, in compact_segments
    for tag, key, offset, data in self.io.iter_objects(segment,
include_data=True):
  File "/usr/lib64/python3.6/site-packages/borg/repository.py", line
1437, in iter_objects
    read_data=read_data)
  File "/usr/lib64/python3.6/site-packages/borg/repository.py", line
1533, in _read
    segment, offset))
borg.helpers.IntegrityError: Data integrity error: Segment entry
checksum mismatch [segment 2438, offset 9906907]

Platform: Linux fqdn 3.10.0-1160.31.1.el7.x86_64 #1 SMP Thu Jun 10
13:32:12 UTC 2021 x86_64
Linux: CentOS Linux 7.9.2009 Core
Borg: 1.1.16  Python: CPython 3.6.8 msgpack: 0.5.6
PID: 25826  CWD: /root
sys.argv: ['/bin/borg', 'create', '--verbose', '--stats', '--list',
'--filter', 'AME', '--show-rc', '--show-version', '--compression',
'zstd', '--exclude-caches', '--exclude', '/mnt/snapbak/root/.caches',
'--exclude', '/mnt/snapbak/home/\\*/.caches', '--exclude',
'/mnt/snapbak/var/cache/\\*', '--exclude', '/mnt/snapbak/var/tmp/\\*',
'--exclude', '/mnt/snapbak/tmp/\\*',
'/mnt/borg/borg-backups/backup.borg::2021-07-18-fqdn-25793-system',
'/mnt/snapbak']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2




-- 

Mason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210726/26e9e306/attachment.html>

From tschoening at am-soft.de  Mon Jul 26 14:26:42 2021
From: tschoening at am-soft.de (=?utf-8?B?VGhvcnN0ZW4gU2Now7ZuaW5n?=)
Date: Mon, 26 Jul 2021 20:26:42 +0200
Subject: [Borgbackup] (no subject)
In-Reply-To: <CAM_s9ZnM1MR4DVvPyAef9b0+ik6zP677SWNyrU74po07jPv0Bg@mail.gmail.com>
References: <CAM_s9ZnM1MR4DVvPyAef9b0+ik6zP677SWNyrU74po07jPv0Bg@mail.gmail.com>
Message-ID: <324556113.20210726202642@am-soft.de>

Guten Tag Mason Schmitt via Borgbackup,
am Montag, 26. Juli 2021 um 20:10 schrieben Sie:

> I had seen a single badblock report for the USB drive, so I replaced the
> USB backup drive and initialized a new borg repo and things seemed to be ok
> for two weeks.

> As an aside - I ran badblocks against the old USB drive and it didn't find
> any issues with the drive.

What file system do you use on USB, something with it's own checksums
like BTRFS or ZFS? That might help to further recognize a hardware vs.
software related problem. Depending on how exactly your USB device
works, remember that cables, controllers etc. can easily break as
well, not only drives within their enclosures, but really the
enclosures themself.

Besides that, in theory you need to even look at the USB controllers,
the interfaces being part of the data path to USB etc. within your
server. Each of those components might simply be damaged for any
reason. But if I was you, I would start with the file system if that
is not already checksumming to see if it barks as well when borg does.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska


F?r R?ckfragen stehe ich Ihnen sehr gerne zur Verf?gung.

Mit freundlichen Gr??en

Thorsten Sch?ning


Tel: 05151 9468 0
Fax: 05151 9468 88
Mobil: 
Webseite: https://www.am-soft.de 

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.?ist ein Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

AM-Soft IT-Service - Bitstore Hameln GmbH i.G.
Brandenburger Str. 7c
31789 Hameln
Tel: 05151 9468 0

Bitstore IT-Consulting GmbH
Zentrale - Berlin Lichtenberg
Frankfurter Allee 285
10317 Berlin
Tel: 030 453 087 80

CBS IT-Service - Bitstore Kaulsdorf UG
Tel: 030 453 087 880 1

B?ro Dallgow-D?beritz
Tel: 03322 507 020

B?ro Kloster Lehnin
Tel: 033207 566 530

PCE IT-Service?- Bitstore Darmstadt UG
Darmstadt
Tel: 06151 392 973 0

B?ro Neuruppin
Tel: 033932 606 090

ACI EDV Systemhaus - Bitstore Dresden GmbH
Dresden
Tel: 0351 254 410

Das Systemhaus - Bitstore Magdeburg GmbH
Magdeburg
Tel: 0391 636 651 0

Allerdata.IT - Bitstore Wittenberg GmbH
Wittenberg
Tel: 03491?876 735 7

B?ro Liebenwalde
Tel: 033054 810 00

HSA - das B?ro - Bitstore Altenburg UG
Altenburg
Tel: 0344 784 390 97

Bitstore IT ? Consulting GmbH
NL Piesteritz 
Piesteritz
Tel: 03491 644 868 6

Solltec IT-Services - Bitstore Braunschweig UG
Braunschweig
Tel: 0531 206 068 0

MF Computer Service - Bitstore G?tersloh GmbH
G?tersloh
Tel: 05245?920 809 3

Firmensitz: AM-Soft IT-Service - Bitstore Hameln GmbH i.G. , Brandenburger Str. 7c , 31789 Hameln
Gesch?ftsf?hrer Janine Galonska







From mason at ftlcomputing.com  Mon Jul 26 14:53:21 2021
From: mason at ftlcomputing.com (Mason Schmitt)
Date: Mon, 26 Jul 2021 11:53:21 -0700
Subject: [Borgbackup] (no subject)
In-Reply-To: <324556113.20210726202642@am-soft.de>
References: <CAM_s9ZnM1MR4DVvPyAef9b0+ik6zP677SWNyrU74po07jPv0Bg@mail.gmail.com>
 <324556113.20210726202642@am-soft.de>
Message-ID: <CAM_s9Zk58cXanKiJE-__8Cgqs2mmFaeO70ML1zkzf-e3iB2MTg@mail.gmail.com>

Hi Thorsten,

On Mon, 26 Jul 2021 at 11:27, Thorsten Sch?ning <tschoening at am-soft.de>
wrote:

> Guten Tag Mason Schmitt via Borgbackup,
> am Montag, 26. Juli 2021 um 20:10 schrieben Sie:
>
> > I had seen a single badblock report for the USB drive, so I replaced the
> > USB backup drive and initialized a new borg repo and things seemed to be
> ok
> > for two weeks.
>
> > As an aside - I ran badblocks against the old USB drive and it didn't
> find
> > any issues with the drive.
>
> What file system do you use on USB, something with it's own checksums
> like BTRFS or ZFS? That might help to further recognize a hardware vs.
> software related problem.


I'm currently using EXT4.  CentOS 7 doesn't support ZFS and I've been
watching BTRFS for years now and it just never seems ready for general
production use, so EXT4 seemed like a good fit for a backup drive.


Depending on how exactly your USB device
> works, remember that cables, controllers etc. can easily break as
> well, not only drives within their enclosures, but really the
> enclosures themself.
>
> Besides that, in theory you need to even look at the USB controllers,
> the interfaces being part of the data path to USB etc. within your
> server. Each of those components might simply be damaged for any
> reason.


Yes, I had considered that it might not be the drive or cable (I've already
replaced the cable too), but rather something on the motherboard.

Do these suggestions imply that the only reasonable answer is that I do
have a hardware problem?



> But if I was you, I would start with the file system if that
> is not already checksumming to see if it barks as well when borg does.
>

I might try BTRFS on this particular backup drive, just to see what
happens.  Thanks for the suggestion.


--
Mason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210726/51b449dd/attachment.html>

From devzero at web.de  Sat Jul 31 03:58:24 2021
From: devzero at web.de (Roland)
Date: Sat, 31 Jul 2021 09:58:24 +0200
Subject: [Borgbackup] (no subject)
In-Reply-To: <CAM_s9Zk58cXanKiJE-__8Cgqs2mmFaeO70ML1zkzf-e3iB2MTg@mail.gmail.com>
References: <CAM_s9ZnM1MR4DVvPyAef9b0+ik6zP677SWNyrU74po07jPv0Bg@mail.gmail.com>
 <324556113.20210726202642@am-soft.de>
 <CAM_s9Zk58cXanKiJE-__8Cgqs2mmFaeO70ML1zkzf-e3iB2MTg@mail.gmail.com>
Message-ID: <c8d8eed4-3719-4889-7b26-c4c22e9c4101@web.de>


>
> I'm currently using EXT4.? CentOS 7 doesn't support ZFS and I've been
> watching BTRFS for years now and it just never seems?ready for general
> production use, so EXT4 seemed like a good fit for a backup drive.


https://btrfs.wiki.kernel.org/index.php/Production_Users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210731/8ac41921/attachment.html>

From tschoening at am-soft.de  Fri Aug 20 12:14:31 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Fri, 20 Aug 2021 18:14:31 +0200
Subject: [Borgbackup] How does BorgBackup handle symlinks?
Message-ID: <91378233.20210820181431@am-soft.de>

Hi everyone,

I'm mounting some directories from a Synology NAS using SSHFS and am
backing files up of that. Some files are symlinks using absolute
paths and reading those results in the following error messages:

> /mnt/[...]/root/bin/btrfsck: readlink: [Errno 2] No such file or directory: '/mnt/[...]/root/bin/btrfsck'
> E /mnt/[...]/root/bin/btrfsck

Of course I simply don't want "readlink" to be called at all, but am
not sure where it comes from. I'm somewhat sure that i DO NOT have
"fowllow_symlinks" enabled when mounting SSHFS. Instead, Synology uses
some own mechanism to provide SFTP using some "internal-sftp" stuff
handling permissions checks etc. as well already. It might be that
those simply "follow_symlinks" by default and I don't know how to
disable that. Providing something like "-o follow_symlinks=no" on my
site doesn't seem to work:

> fuse: unknown option(s): `-o follow_symlinks=no'

So, from my understand of the path above, as the symlink is the last
part of the path, BorgBackup should try to read it as-is, correct? It
doesn't trigger something like "readlink" on its own? Or is there some
option to explicitly disable resolving of symlinks, just to be sure?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From richjunk at pacbell.net  Sat Aug 21 14:24:34 2021
From: richjunk at pacbell.net (Richard)
Date: Sat, 21 Aug 2021 11:24:34 -0700
Subject: [Borgbackup] BorgBackup Cache Not Regenerating
References: <47f9481d-1f39-d68d-88f6-a76975c1ca24.ref@pacbell.net>
Message-ID: <47f9481d-1f39-d68d-88f6-a76975c1ca24@pacbell.net>


I am running BorgBackup (1.1.17) as root on two Linux systems.  Each
system has 64GB ram and is not rebooted often so I have most users
(including root) ~/.cache directory pointed at a tmpfs (ramdisk) file
system which means the cache, including BorgBackup's cache is deleted on
reboot.

On one system after a reboot, "borg create" runs as expected except it
takes much longer to create an archive as it needs to regenerate the
~/.cache/borg cache.  The next time BorgBackup runs on this system it is
'fast' as normal.

On the 2nd system BorgBackup (again run as root) never seems to generate
a cache in ~/.cache/borg.  I do not see any errors in the BorgBackup
output.  Any idea why?  Is there some setting to tell Borg where to put
it's cache? Is there some verbose mode I should be using to see what the
issue is?

I am new to BorgBackup and if this cache issue is addressed in the
BorgBackup documentation please let me know where.

I am using a script to launch borg based on a sample in the
documentation.  Here is part of my borg create script:


borg create                         \
    --stats                         \
    --show-rc                       \
    --one-file-system               \
    --noatime                       \
    --noctime                       \
    --nobsdflags                    \
    --compression=auto,zstd,10      \
    --chunker-params=12,23,16,4095  \
    --exclude-caches                \
    --exclude '/home/*/.cache'      \
    --exclude '/root/.cache'        \
    --exclude '/var/tmp/*'          \
    --exclude '/tmp'                \
                                    \
                                    \
    ::'{hostname}-{now}'            \
    /etc                            \
    /root                           \
    /var/lib                        \
    /boot                           \
    /lib/modules                    \
    /usr/local                      \
    /home                           \
    /var/log                        \
    /data0/                         \
    /data1/                         \
....


Thanks.

From tschoening at am-soft.de  Sat Aug 21 16:51:34 2021
From: tschoening at am-soft.de (=?windows-1250?Q?Thorsten_Sch=F6ning?=)
Date: Sat, 21 Aug 2021 22:51:34 +0200
Subject: [Borgbackup] BorgBackup Cache Not Regenerating
In-Reply-To: <47f9481d-1f39-d68d-88f6-a76975c1ca24@pacbell.net>
References: <47f9481d-1f39-d68d-88f6-a76975c1ca24.ref@pacbell.net>
 <47f9481d-1f39-d68d-88f6-a76975c1ca24@pacbell.net>
Message-ID: <190766328.20210821225134@am-soft.de>

Guten Tag Richard,
am Samstag, 21. August 2021 um 20:24 schrieben Sie:

> On the 2nd system BorgBackup (again run as root) never seems to generate
> a cache in ~/.cache/borg.  I do not see any errors in the BorgBackup
> output.  Any idea why?

Increase log level to see if Borg doesn't do things without running
into any warning or error condition.

https://borgbackup.readthedocs.io/en/stable/usage/general.html#logging

Besides that: How do you remount or whatever ~/.cache exactly? There
might be some chicken-egg-probelm with when you execute Borg. Is
~/.cache created ever at all for the user for which Borg doesn't seem
to create it? Is that directory maybe availble in the default file
system instead of youre expected TMPFS?

> Is there some setting to tell Borg where to put
> it's cache?

Have a look at the environment variables:

> BORG_BASE_DIR
> BORG_CACHE_DIR

https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables

> I am using a script to launch borg based on a sample in the
> documentation.  Here is part of my borg create script:

Just for your interest, as I'm new to Borg as well and found the
following pretty interesting/useful to avoid custom scripts for
execution itself.

https://torsion.org/borgmatic/

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From richjunk at pacbell.net  Mon Aug 23 23:07:36 2021
From: richjunk at pacbell.net (Richard)
Date: Mon, 23 Aug 2021 20:07:36 -0700
Subject: [Borgbackup] BorgBackup Cache Not Regenerating
In-Reply-To: <6a017d09-48cf-409f-1ab4-a5cd86b74c32@pacbell.net>
References: <6a017d09-48cf-409f-1ab4-a5cd86b74c32@pacbell.net>
Message-ID: <7fa73f1c-e965-44df-ed8c-680b7c42d902@pacbell.net>



> Increase log level to see if Borg doesn't do things without running
> into any warning or error condition.

I set 'borg create' to --info.   I did not see any issues, however, this
time the cache was created.

> Besides that: How do you remount or whatever ~/.cache exactly? There
> might be some chicken-egg-probelm with when you execute Borg. Is
> ~/.cache created ever at all for the user for which Borg doesn't seem
> to create it? Is that directory maybe availble in the default file
> system instead of youre expected TMPFS?

The issue might have been the chicken-or-egg.  I use xterm for most
things and when I su up to root /root/.bash_profile runs:

# for root cache area
if [ ! -d /tmp/cache-root ]; then
        /bin/mkdir /tmp/cache-root
        /bin/chmod 700 /tmp/cache-root
fi

and /root/.cache is a symbolic link to /tmp/cache-root.

/tmp is a tmpfs file system. So the first time a root bash shell is open
the above code would create the cache area.  I thought that the bash
shell script containing 'borg create' would also call
/root/.bash_profile, but perhaps not?  If I reboot the system and don't
open a root shell before 'borg create' is run there may not be a cache
area.   I just added the code above to the top of my borg backup script
 to ensure the cache area exists.  Now after the next reboot  I'll see
if the issue is resolved.

Thank you for the reply.



From ceo.teo.en.ming at gmail.com  Sat Aug 28 06:23:53 2021
From: ceo.teo.en.ming at gmail.com (Turritopsis Dohrnii Teo En Ming)
Date: Sat, 28 Aug 2021 18:23:53 +0800
Subject: [Borgbackup] We need excellent and well-written documentation for
 linux backup software
Message-ID: <CAMEJMGGHxM6-Kp50B8mY6aN8a5Ujsu-_DxetouNSz8cfmgRdGA@mail.gmail.com>

Subject: We need excellent and well-written documentation for linux
backup software

Good day from Singapore,

Our company, which is a Systems Integrator (SI) in Singapore, is
evaluating more than 30 different backup software for linux systems.

We need excellent and well-written documentation for your linux backup
software. Please provide us with links on how to setup your linux
backup software.

We are looking forward to your replies. Thank you very much for your
kind assistance.

Mr. Turritopsis Dohrnii Teo En Ming, 43 years old as of 28 August
2021, is a TARGETED INDIVIDUAL living in Singapore. He is an IT
Consultant with a System Integrator (SI)/computer firm in Singapore.
He is an IT enthusiast.




-----BEGIN EMAIL SIGNATURE-----

The Gospel for all Targeted Individuals (TIs):

[The New York Times] Microwave Weapons Are Prime Suspect in Ills of
U.S. Embassy Workers

Link:
https://www.nytimes.com/2018/09/01/science/sonic-attack-cuba-microwave.html

********************************************************************************************

Singaporean Targeted Individual Mr. Turritopsis Dohrnii Teo En Ming's
Academic Qualifications as at 14 Feb 2019 and refugee seeking attempts
at the United Nations Refugee Agency Bangkok (21 Mar 2017), in Taiwan
(5 Aug 2019) and Australia (25 Dec 2019 to 9 Jan 2020):

[1] https://tdtemcerts.wordpress.com/

[2] https://tdtemcerts.blogspot.sg/

[3] https://www.scribd.com/user/270125049/Teo-En-Ming

-----END EMAIL SIGNATURE-----

From borgbackup at aluaces.fastmail.com  Sat Aug 28 06:45:54 2021
From: borgbackup at aluaces.fastmail.com (Alberto Luaces)
Date: Sat, 28 Aug 2021 12:45:54 +0200
Subject: [Borgbackup] We need excellent and well-written documentation
 for linux backup software
In-Reply-To: <CAMEJMGGHxM6-Kp50B8mY6aN8a5Ujsu-_DxetouNSz8cfmgRdGA@mail.gmail.com>
References: <CAMEJMGGHxM6-Kp50B8mY6aN8a5Ujsu-_DxetouNSz8cfmgRdGA@mail.gmail.com>
Message-ID: <87mtp1etjx.fsf@pantallon>


Turritopsis Dohrnii Teo En Ming writes:
> We need excellent and well-written documentation for your linux backup
> software. Please provide us with links on how to setup your linux
> backup software.

Please read https://borgbackup.readthedocs.io/en/stable/ , specially https://borgbackup.readthedocs.io/en/stable/authors.html#license

From sitaramc at gmail.com  Sun Aug 29 00:24:22 2021
From: sitaramc at gmail.com (Sitaram Chamarty)
Date: Sun, 29 Aug 2021 09:54:22 +0530
Subject: [Borgbackup] hundreds of 17-byte files (1744 files out of 1762)
Message-ID: <YSsL9pmfuoTOJ4OP@sita-dell>

Hi

I'm trying to use `par2` on a borg repository that I am
archiving long term; par2 uses Reed-Solomon coding to produce
extra files that help recreate the data as long as it's not too
damaged.

My borg repo is 3646 MB.  It has 1762 files, of which 1744 are
17-byte files with exactly the same content.  This seems to
completely throw off par2 in terms of performance.

I've asked the par2 folks also, but I thought it would be worth
asking here: why do all those files have exactly the same
content, and if so why are ther 1744 of them?

thanks!
sitaram

PS: Just FYI, some numbers:

    straight par2 on the borg repo:

        3851.39s u 3.65s s 702% 9:08.45 | 0 sw 0 maj 36063 min | 0 txt 0 dat 141 max

    If I tar up the repo and run par2 on the single tar file:

        516.79s u 1.81s s 601% 1:26.17 | 0 sw 0 maj 33887 min | 0 txt 0 dat 134 max

    Similarly, if I run it only on files not equal to 17 bytes

        500.60s u 1.73s s 679% 1:13.88 | 0 sw 1 maj 34525 min | 0 txt 0 dat 135 max

From tw at waldmann-edv.de  Sun Aug 29 12:28:22 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Sun, 29 Aug 2021 18:28:22 +0200
Subject: [Borgbackup] hundreds of 17-byte files (1744 files out of 1762)
In-Reply-To: <YSsL9pmfuoTOJ4OP@sita-dell>
References: <YSsL9pmfuoTOJ4OP@sita-dell>
Message-ID: <9208fb7e-1528-ac38-988d-97436edb7881@thinkmo.de>


> I'm trying to use `par2` on a borg repository that I am
> archiving long term; par2 uses Reed-Solomon coding to produce
> extra files that help recreate the data as long as it's not too
> damaged.
>
> My borg repo is 3646 MB. It has 1762 files, of which 1744 are
> 17-byte files with exactly the same content. This seems to
> completely throw off par2 in terms of performance.

If a few small files "throw off" par2 in terms of performance, what 
would be the case if you had a really big backup?

> I've asked the par2 folks also, but I thought it would be worth
> asking here: why do all those files have exactly the same
> content, and if so why are ther 1744 of them?

This is a (harmless) bug in borg 1.1.x, see the issue tracker.

borg 1.2 will have a cleanup action to remove the superfluous commit files.




From sitaramc at gmail.com  Tue Aug 31 12:02:55 2021
From: sitaramc at gmail.com (Sitaram Chamarty)
Date: Tue, 31 Aug 2021 21:32:55 +0530
Subject: [Borgbackup] hundreds of 17-byte files (1744 files out of 1762)
In-Reply-To: <9208fb7e-1528-ac38-988d-97436edb7881@thinkmo.de>
References: <YSsL9pmfuoTOJ4OP@sita-dell>
 <9208fb7e-1528-ac38-988d-97436edb7881@thinkmo.de>
Message-ID: <YS5Sr0hDVymPv1LW@sita-dell>

On Sun, Aug 29, 2021 at 06:28:22PM +0200, Thomas Waldmann wrote:
> 
> > I'm trying to use `par2` on a borg repository that I am
> > archiving long term; par2 uses Reed-Solomon coding to produce
> > extra files that help recreate the data as long as it's not too
> > damaged.
> > 
> > My borg repo is 3646 MB. It has 1762 files, of which 1744 are
> > 17-byte files with exactly the same content. This seems to
> > completely throw off par2 in terms of performance.
> 
> If a few small files "throw off" par2 in terms of performance, what would be
> the case if you had a really big backup?

Large files work fine -- performance scales much more linearly
with overall size of data.  It's some block size thing that
causes extremely small files to be a problem.

If I understand correctly, the two other popular tools that use
Reed Solomon coding have similar issues though their base speed
may be better, so this appears to be inherent to the algorithm,
not a fault in this particular tool.

(And I don't know if I'd call 1744 files "few".  Not when they
constitute almost 99% of the files in the repo; there are only
18 others).

> 
> > I've asked the par2 folks also, but I thought it would be worth
> > asking here: why do all those files have exactly the same
> > content, and if so why are ther 1744 of them?
> 
> This is a (harmless) bug in borg 1.1.x, see the issue tracker.

couldn't find an issue that is exactly about this.  #5679 and
#5315 seem closest -- is it one of those?

> 
> borg 1.2 will have a cleanup action to remove the superfluous commit files.

good to know.

meanwhile it's trivial for me to tar up all the 17-byte files,
then give `par2` only files that are not 17 bytes long.  I can
always untar and extract them if a repair is needed.


From tw at waldmann-edv.de  Wed Sep  1 09:00:53 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Wed, 1 Sep 2021 15:00:53 +0200
Subject: [Borgbackup] hundreds of 17-byte files (1744 files out of 1762)
In-Reply-To: <YS5Sr0hDVymPv1LW@sita-dell>
References: <YSsL9pmfuoTOJ4OP@sita-dell>
 <9208fb7e-1528-ac38-988d-97436edb7881@thinkmo.de>
 <YS5Sr0hDVymPv1LW@sita-dell>
Message-ID: <a955abbf-22ff-fa18-0ee7-0e3e3b5cb74e@thinkmo.de>


>>> My borg repo is 3646 MB. It has 1762 files, of which 1744 are
>>> 17-byte files with exactly the same content. This seems to
>>> completely throw off par2 in terms of performance.
>> If a few small files "throw off" par2 in terms of performance, what 
>> would be
>> the case if you had a really big backup?
> Large files work fine -- performance scales much more linearly
> with overall size of data. It's some block size thing that
> causes extremely small files to be a problem.
>
> If I understand correctly, the two other popular tools that use
> Reed Solomon coding have similar issues though their base speed
> may be better, so this appears to be inherent to the algorithm,
> not a fault in this particular tool.
>
> (And I don't know if I'd call 1744 files "few". Not when they
> constitute almost 99% of the files in the repo; there are only
> 18 others).

Well, you could also have 2000 big files in your borg backend - your 
3.5GB repo isn't very big yet.

So if you have a problem with the small files now, I guess you would 
have a bigger problem if they were bigger.

With 500MB (default) large segment files that would already be 1TB, so 
relatively large overall.

But you have to try whether 500MB is the best segment size for you 
anyway, for some scenarios some other size between 10 and 500MB might be 
better.

>> This is a (harmless) bug in borg 1.1.x, see the issue tracker.
> couldn't find an issue that is exactly about this. #5679 and
> #5315 seem closest -- is it one of those?

https://github.com/borgbackup/borg/issues/2850


> meanwhile it's trivial for me to tar up all the 17-byte files,
> then give `par2` only files that are not 17 bytes long. I can
> always untar and extract them if a repair is needed.
>
If you remove the commits there might be data loss if borg thinks all 
inside that repo is uncommitted crap.




From sitaramc at gmail.com  Wed Sep  1 10:03:17 2021
From: sitaramc at gmail.com (Sitaram Chamarty)
Date: Wed, 1 Sep 2021 19:33:17 +0530
Subject: [Borgbackup] hundreds of 17-byte files (1744 files out of 1762)
In-Reply-To: <a955abbf-22ff-fa18-0ee7-0e3e3b5cb74e@thinkmo.de>
References: <YSsL9pmfuoTOJ4OP@sita-dell>
 <9208fb7e-1528-ac38-988d-97436edb7881@thinkmo.de>
 <YS5Sr0hDVymPv1LW@sita-dell>
 <a955abbf-22ff-fa18-0ee7-0e3e3b5cb74e@thinkmo.de>
Message-ID: <YS+IJS7y1yT4PMIi@sita-dell>

On Wed, Sep 01, 2021 at 03:00:53PM +0200, Thomas Waldmann wrote:
> 
> > > > My borg repo is 3646 MB. It has 1762 files, of which 1744 are
> > > > 17-byte files with exactly the same content. This seems to
> > > > completely throw off par2 in terms of performance.
> > > If a few small files "throw off" par2 in terms of performance, what
> > > would be
> > > the case if you had a really big backup?
> > Large files work fine -- performance scales much more linearly
> > with overall size of data. It's some block size thing that
> > causes extremely small files to be a problem.
> > 
> > If I understand correctly, the two other popular tools that use
> > Reed Solomon coding have similar issues though their base speed
> > may be better, so this appears to be inherent to the algorithm,
> > not a fault in this particular tool.
> > 
> > (And I don't know if I'd call 1744 files "few". Not when they
> > constitute almost 99% of the files in the repo; there are only
> > 18 others).
> 
> Well, you could also have 2000 big files in your borg backend - your 3.5GB
> repo isn't very big yet.

I already said large files work fine -- performance is
commensurate with the overall data size, and scales as expected.

Small files is where par2 breaks down.

> So if you have a problem with the small files now, I guess you would have a
> bigger problem if they were bigger.

Not at all.  Or at least not in a way that would be
surprising or unexpected.

> With 500MB (default) large segment files that would already be 1TB, so
> relatively large overall.
> 
> But you have to try whether 500MB is the best segment size for you anyway,
> for some scenarios some other size between 10 and 500MB might be better.
> 
> > > This is a (harmless) bug in borg 1.1.x, see the issue tracker.
> > couldn't find an issue that is exactly about this. #5679 and
> > #5315 seem closest -- is it one of those?
> 
> https://github.com/borgbackup/borg/issues/2850

thanks.

> > meanwhile it's trivial for me to tar up all the 17-byte files,
> > then give `par2` only files that are not 17 bytes long. I can
> > always untar and extract them if a repair is needed.
> > 
> If you remove the commits there might be data loss if borg thinks all inside
> that repo is uncommitted crap.

who said I'm removing them?

Par2 protects against bit flips etc., on disk.  When I mount a
disk some months later, I'd run par2 verify.

Since I did not *create* par2 files for the one thousand seven
hundred forty four 17 byte files, any errors in them won't get
verified.

But there is a tar file that contains all of them -- if that tar
file gets verified (and repaired if needed), I untar it to
overwrite clean copies of those 17-byte files.  The ones on disk
may have been good, or may not; I simply overwrite them.

Then the repo is back to what it was.



From ernest.chiarello at univ-fcomte.fr  Thu Sep  2 05:59:30 2021
From: ernest.chiarello at univ-fcomte.fr (Ernest Chiarello)
Date: Thu, 2 Sep 2021 11:59:30 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
Message-ID: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>

Hi all,

i am new in that list... and I am using borgbackup to back up data 
managed by libvirt / kvm.

i want to backup /var/lib/libvirt/backups/onlyoffice folder with 
borgbackup, it has some xml files and some snapshots (.bimg), as well as 
qcow2 image.

the image weighs 14 GB.

|root at kvm610:~ # ls -lh 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.qcow2 -rwxr-xr-x 1 
libvirt-qemu kvm 14G avril 12 07:17 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.qcow2* |

the first borgbackup backup lasts 2m37, all the files are tagged A, 
because "added".

|root at kvm610:~ # time borg create --list 
/var/lib/libvirt/borgbackup/kvm610::trial4 
/var/lib/libvirt/backups/onlyoffice A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-28_06:25:22.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-23_06:25:35.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-02_06:25:03.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-26_06:25:35.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-29_06:25:14.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-30_06:25:17.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-31_06:25:26.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-01_06:25:10.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-22_06:25:08.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-24_06:25:51.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-25_06:25:23.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-27_06:25:14.xml A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-141702 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-131701 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-121701 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-111701 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-101701 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-091701 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-081701 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-071702 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.qcow2 d 
/var/lib/libvirt/backups/onlyoffice real 2m37,093s user 1m46,503s sys 
0m11,726s |

I then restart the same backup, in the same warehouse but with a 
different archive name (trial5). all the xml and bimg files are tagged 
U, like "updatedb", but the qcow2 image is still tagged A, even though 
it hasn't moved ... the VM has been shut down since April.

|root at kvm610:~ # time borg create --list 
/var/lib/libvirt/borgbackup/kvm610::trial5 
/var/lib/libvirt/backups/onlyoffice U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-28_06:25:22.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-23_06:25:35.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-02_06:25:03.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-26_06:25:35.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-29_06:25:14.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-30_06:25:17.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-31_06:25:26.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-01_06:25:10.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-22_06:25:08.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-24_06:25:51.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-25_06:25:23.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-27_06:25:14.xml U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-141702 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-131701 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-121701 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-111701 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-101701 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-091701 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-081701 U 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-071702 A 
/var/lib/libvirt/backups/onlyoffice/onlyoffice.qcow2 d 
/var/lib/libvirt/backups/onlyoffice real 2m36,473s user 1m45,359s sys 
0m11,831s |

you have an idea to explain this?

and the backup time is the same ... 2m37 / 2m36

thanks in advance,


Ernest.

-- 
Ernest CHIARELLO - Ernest.Chiarello at univ-fcomte.fr

UMR6049 Th?MA - CNRS / Universit? de Franche-Comt?
Service de gestion et production de l'information
32 rue M?gevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

USR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210902/6b352f8b/attachment.html>

From tschoening at am-soft.de  Thu Sep  2 07:01:05 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Thu, 2 Sep 2021 13:01:05 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
Message-ID: <647998900.20210902130105@am-soft.de>

Guten Tag Ernest Chiarello,
am Donnerstag, 2. September 2021 um 11:59 schrieben Sie:

> I then restart the same backup, in the same warehouse but with a
> different archive name (trial5). all the xml and bimg files are
> tagged U, like "updatedb", but the qcow2 image is still tagged A,
> even though it hasn't moved ... the VM has been shut down since April.

Sounds like the following, so carefully look at file time stamps:

> The files cache is used to determine whether Borg already ?knows? /
> has backed up a file and if so, to skip the file from chunking. It
> intentionally excludes files that have a timestamp which is the same
> as the newest timestamp in the created archive.

You can make sure by creating an additional, small file in the backup
source folder:

> If you want to avoid unnecessary chunking, just create or touch a
> small or empty file in your backup source file set (so that one has
> the latest timestamp, not your 50GB VM disk image) and, if you do
> snapshots, do the snapshot after that.

https://borgbackup.readthedocs.io/en/stable/faq.html#i-am-seeing-a-added-status-for-an-unchanged-file

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From ernest.chiarello at univ-fcomte.fr  Thu Sep  2 09:08:13 2021
From: ernest.chiarello at univ-fcomte.fr (Ernest Chiarello)
Date: Thu, 2 Sep 2021 15:08:13 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <647998900.20210902130105@am-soft.de>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
Message-ID: <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>

Vielen Dank, Thorsten,

I'm not sure I fully understand what is written in the documentation... 
but i tried to add a new small file in the archive, as suggested. ?

Le 02.09.21 ? 13:01, Thorsten Sch?ning a ?crit?:
> You can make sure by creating an additional, small file in the backup
> source folder:
>
>> If you want to avoid unnecessary chunking, just create or touch a
>> small or empty file in your backup source file set (so that one has
>> the latest timestamp, not your 50GB VM disk image) and, if you do
>> snapshots, do the snapshot after that.

the new added file is passwd... the first backup (with passwd) takes the 
same time as the previous one: 2m34

root at kvm610:/var/lib/libvirt/backups/onlyoffice # time borg create 
--list /var/lib/libvirt/borgbackup/kvm610::essai6 
/var/lib/libvirt/backups/onlyoffice

U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-28_06:25:22.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-23_06:25:35.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-02_06:25:03.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-26_06:25:35.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-29_06:25:14.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-30_06:25:17.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-31_06:25:26.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-01_06:25:10.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-22_06:25:08.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-24_06:25:51.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-25_06:25:23.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-27_06:25:14.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-141702
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-131701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-121701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-111701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-101701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-091701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-081701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-071702
A /var/lib/libvirt/backups/onlyoffice/onlyoffice.qcow2
A /var/lib/libvirt/backups/onlyoffice/passwd
d /var/lib/libvirt/backups/onlyoffice

real??? 2m34,957s
user??? 1m44,797s
sys??? 0m12,468s

but the next backup is immediate: 0m07 !!

root at kvm610:/var/lib/libvirt/backups/onlyoffice # time borg create 
--list /var/lib/libvirt/borgbackup/kvm610::essai7 
/var/lib/libvirt/backups/onlyoffice
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-28_06:25:22.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-23_06:25:35.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-02_06:25:03.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-26_06:25:35.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-29_06:25:14.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-30_06:25:17.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-31_06:25:26.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-09-01_06:25:10.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-22_06:25:08.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-24_06:25:51.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-25_06:25:23.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice_2021-08-27_06:25:14.xml
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-141702
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-131701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-121701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-111701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-101701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-091701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-081701
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.bimg-20210412-071702
U /var/lib/libvirt/backups/onlyoffice/onlyoffice.qcow2
A /var/lib/libvirt/backups/onlyoffice/passwd
d /var/lib/libvirt/backups/onlyoffice

real??? 0m0,682s
user??? 0m0,454s
sys??? 0m0,146s

and the qcow2 file is tagged "U", as expected.

but I don't see how to avoid this type of problem when saving large 
files from time to time (qcow2 images)... ?

What do you think ?


Ernest.

> https://borgbackup.readthedocs.io/en/stable/faq.html#i-am-seeing-a-added-status-for-an-unchanged-file
>
> Mit freundlichen Gr??en
>
> Thorsten Sch?ning
>
-- 
Ernest CHIARELLO - Ernest.Chiarello at univ-fcomte.fr

UMR6049 Th?MA - CNRS / Universit? de Franche-Comt?
Service de gestion et production de l'information
32 rue M?gevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

USR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210902/b73de7e8/attachment-0001.html>

From tschoening at am-soft.de  Thu Sep  2 09:48:48 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Thu, 2 Sep 2021 15:48:48 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
Message-ID: <14910591148.20210902154848@am-soft.de>

Guten Tag Ernest Chiarello,
am Donnerstag, 2. September 2021 um 15:08 schrieben Sie:

> but I don't see how to avoid this type of problem when saving large
> files from time to time (qcow2 images)... ?

Things depend on if you ONLY store the large failes additionally and
never more current smaller ones as well. For most backup purposes this
is not the case, as some smaller files will most likely be available,
not only large images. If it's different in your case, you simply need
to make sure by creating some small dummy file to "trick" BorgBackup.

I did exactly that e.g. for VMs which I archive and don't change
anymore: VM-images are at timestamp X, dummy file is at timestamp X+1,
so no problem ever. If VM-images would change at some point again, one
would simply touch the dummy file again so that it's newer and things
work as before. My dummy fille even simply contains the content of the
linked FAQ, so that I always know why it's there. :-)

If you want to make things easier to use, I can suggest BorgMatic.
That allows you to have everythign in yome YAML-config and
additionally provides hooks being executed before the backup. Those
hooks can simply be used to touch your dummy file always.

> hooks:
>     # Custom preparation scripts to run.
>     before_backup:
>         - prepare-for-backup.sh

https://torsion.org/borgmatic/

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From Ernest.Chiarello at univ-fcomte.fr  Fri Sep  3 09:32:07 2021
From: Ernest.Chiarello at univ-fcomte.fr (Ernest CHIARELLO)
Date: Fri, 3 Sep 2021 15:32:07 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <14910591148.20210902154848@am-soft.de>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
Message-ID: <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>

Hello Thorsten,

thank you for taking the time to answer me ! ?

I want to clarify my request. we are typically in a case of libvirt/kvm 
virtualization.

for the moment, i was using borgbackup in order to save VM-images and 
snaphots from a local backups folder to a repository on a remote 
server... it works perfectly.

actually, VM-images are managed by fi-backup 
<https://github.com/dguerri/LibVirtKvm-scripts/blob/master/fi-backup.sh>, 
a bash script which produces one snaphot per hour of each VM, and which 
consolidates them at the end of the day into a single qcow2 file ... 
then starts again the next day.

after each snaphot creation, fi-backup copies the file to the backups 
folder.when it consolidates all the snapshots (small files), it also 
copies the new image (big file) to the backups folder.it uses the 
command "cp -u", which is expensive in time...

my question is: would it be possible to replace the "cp -u" by "borg 
create" (from /var/lib/libvirt/images to /var/lib/libvirt/backups), in 
order to benefit from the deduplication and to shorten the copy times ?

the creation of dummy files would then also be managed by fi-backup.

What do you think ?

Thanks in advance,

Ernest.


Le 02/09/2021 ? 15:48, Thorsten Sch?ning a ?crit?:
> Guten Tag Ernest Chiarello,
> am Donnerstag, 2. September 2021 um 15:08 schrieben Sie:
>
>> but I don't see how to avoid this type of problem when saving large
>> files from time to time (qcow2 images)... ?
> Things depend on if you ONLY store the large failes additionally and
> never more current smaller ones as well. For most backup purposes this
> is not the case, as some smaller files will most likely be available,
> not only large images. If it's different in your case, you simply need
> to make sure by creating some small dummy file to "trick" BorgBackup.
>
> I did exactly that e.g. for VMs which I archive and don't change
> anymore: VM-images are at timestamp X, dummy file is at timestamp X+1,
> so no problem ever. If VM-images would change at some point again, one
> would simply touch the dummy file again so that it's newer and things
> work as before. My dummy fille even simply contains the content of the
> linked FAQ, so that I always know why it's there. :-)
>
> If you want to make things easier to use, I can suggest BorgMatic.
> That allows you to have everythign in yome YAML-config and
> additionally provides hooks being executed before the backup. Those
> hooks can simply be used to touch your dummy file always.
>
>> hooks:
>>      # Custom preparation scripts to run.
>>      before_backup:
>>          - prepare-for-backup.sh
> https://torsion.org/borgmatic/
>
> Mit freundlichen Gr??en
>
> Thorsten Sch?ning
>

-- 
Ernest CHIARELLO - Ernest.Chiarello at univ-fcomte.fr

UMR6049 Th?MA - CNRS / Universit? de Franche-Comt?
Service de gestion et production de l'information
32 rue M?gevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

UAR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210903/2159761a/attachment.html>

From lazyvirus at gmx.com  Fri Sep  3 11:01:37 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Fri, 3 Sep 2021 17:01:37 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
Message-ID: <20210903170137.6d0a5f31@msi.defcon1.lan>

On Fri, 3 Sep 2021 15:32:07 +0200
Ernest CHIARELLO <Ernest.Chiarello at univ-fcomte.fr> wrote:

Hi Ernest,

> thank you for taking the time to answer me ! ?
> 
> I want to clarify my request. we are typically in a case of
> libvirt/kvm virtualization.
> 
> for the moment, i was using borgbackup in order to save VM-images and 
> snaphots from a local backups folder to a repository on a remote 
> server... it works perfectly.
> 
> actually, VM-images are managed by fi-backup 
> <https://github.com/dguerri/LibVirtKvm-scripts/blob/master/fi-backup.sh>, 
> a bash script which produces one snaphot per hour of each VM, and
> which consolidates them at the end of the day into a single qcow2
> file ... then starts again the next day.

Of course, it entirely depends on what you do inside your VMs, but just
one day of snapshots with a reset every evening seems a bit short.

IF there are files that are users (or admins) modified in these VMs, you
can be sure that one day or another, they'll goof just before a long WE
without noticing it at once?

> after each snaphot creation, fi-backup copies the file to the backups 
> folder.when it consolidates all the snapshots (small files), it also 
> copies the new image (big file) to the backups folder.it uses the 
> command "cp -u", which is expensive in time...
> 
> my question is: would it be possible to replace the "cp -u" by "borg 
> create" (from /var/lib/libvirt/images to /var/lib/libvirt/backups), in 
> order to benefit from the deduplication and to shorten the copy times ?

Is there a very good justification to make your backups from copies
instead of originals ?

Jean-Yves

From Ernest.Chiarello at univ-fcomte.fr  Fri Sep  3 11:32:23 2021
From: Ernest.Chiarello at univ-fcomte.fr (Ernest CHIARELLO)
Date: Fri, 3 Sep 2021 17:32:23 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <20210903170137.6d0a5f31@msi.defcon1.lan>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
 <20210903170137.6d0a5f31@msi.defcon1.lan>
Message-ID: <72e6c6f7-bffa-8760-c828-8a089e766b78@univ-fcomte.fr>

Hi jean-Yves,

Le 03/09/2021 ? 17:01, Bzzzz a ?crit?:
> Is there a very good justification to make your backups from copies
> instead of originals ?

i suppose you can not save the qcow2 file of a running VM. you must stop 
it before.

with fi-backup, the "running" snaphot is in /var/lib/libvirt/images et 
not yet in /var/lib/libvirt/backups/glpi.

example at 17h30:

root at kvm1630:~ # ls -ltr /var/lib/libvirt/images/glpi.*
-rwxr-xr-x 1 libvirt-qemu kvm 12483624960 sept.? 3 07:30 
/var/lib/libvirt/images/glpi.qcow2*
-rw------- 1 libvirt-qemu kvm?? 133431296 sept.? 3 08:17 
/var/lib/libvirt/images/glpi.bimg-20210903-073018
-rw------- 1 libvirt-qemu kvm?? 146210816 sept.? 3 09:17 
/var/lib/libvirt/images/glpi.bimg-20210903-081714
-rw------- 1 libvirt-qemu kvm?? 101908480 sept.? 3 10:17 
/var/lib/libvirt/images/glpi.bimg-20210903-091707
-rw------- 1 libvirt-qemu kvm??? 99418112 sept.? 3 11:17 
/var/lib/libvirt/images/glpi.bimg-20210903-101707
-rw------- 1 libvirt-qemu kvm?? 141361152 sept.? 3 12:17 
/var/lib/libvirt/images/glpi.bimg-20210903-111708
-rw------- 1 libvirt-qemu kvm?? 133955584 sept.? 3 13:17 
/var/lib/libvirt/images/glpi.bimg-20210903-121708
-rw------- 1 libvirt-qemu kvm?? 161742848 sept.? 3 14:17 
/var/lib/libvirt/images/glpi.bimg-20210903-131709
-rw------- 1 libvirt-qemu kvm?? 141885440 sept.? 3 15:17 
/var/lib/libvirt/images/glpi.bimg-20210903-141708
-rw------- 1 libvirt-qemu kvm?? 127205376 sept.? 3 16:17 
/var/lib/libvirt/images/glpi.bimg-20210903-151708
-rw------- 1 libvirt-qemu kvm?? 114950144 sept.? 3 17:17 
/var/lib/libvirt/images/glpi.bimg-20210903-161708
-rw------- 1 libvirt-qemu kvm??? 74711040 sept.? 3 17:30 
/var/lib/libvirt/images/glpi.bimg-20210903-171709

root at kvm1630:~ # ls -ltr /var/lib/libvirt/backups/glpi/
-rwxr-xr-x 1 root root 12483624960 sept.? 3 07:32 glpi.qcow2*
-rw------- 1 root root?? 133431296 sept.? 3 08:17 glpi.bimg-20210903-073018
-rw------- 1 root root?? 146210816 sept.? 3 09:17 glpi.bimg-20210903-081714
-rw------- 1 root root?? 101908480 sept.? 3 10:17 glpi.bimg-20210903-091707
-rw------- 1 root root??? 99418112 sept.? 3 11:17 glpi.bimg-20210903-101707
-rw------- 1 root root?? 141361152 sept.? 3 12:17 glpi.bimg-20210903-111708
-rw------- 1 root root?? 133955584 sept.? 3 13:17 glpi.bimg-20210903-121708
-rw------- 1 root root?? 161742848 sept.? 3 14:17 glpi.bimg-20210903-131709
-rw------- 1 root root?? 141885440 sept.? 3 15:17 glpi.bimg-20210903-141708
-rw------- 1 root root?? 127205376 sept.? 3 16:17 glpi.bimg-20210903-151708
-rw------- 1 root root?? 114950144 sept.? 3 17:17 glpi.bimg-20210903-161708

thanks,

Ernest.

-- 
Ernest CHIARELLO - Ernest.Chiarello at univ-fcomte.fr

UMR6049 Th?MA - CNRS / Universit? de Franche-Comt?
Service de gestion et production de l'information
32 rue M?gevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

UAR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210903/4e207f7e/attachment-0001.html>

From lazyvirus at gmx.com  Fri Sep  3 11:55:22 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Fri, 3 Sep 2021 17:55:22 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <72e6c6f7-bffa-8760-c828-8a089e766b78@univ-fcomte.fr>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
 <20210903170137.6d0a5f31@msi.defcon1.lan>
 <72e6c6f7-bffa-8760-c828-8a089e766b78@univ-fcomte.fr>
Message-ID: <20210903175522.6aaa3c63@msi.defcon1.lan>

On Fri, 3 Sep 2021 17:32:23 +0200
Ernest CHIARELLO <Ernest.Chiarello at univ-fcomte.fr> wrote:

[sorry for the PM, so called "better" management of ML by changing the
answer policy in most of them is in fact a real mess:/]

> Hi jean-Yves,
> 
> Le 03/09/2021 ? 17:01, Bzzzz a ?crit?:  
> > Is there a very good justification to make your backups from copies
> > instead of originals ?  
> 
> i suppose you can not save the qcow2 file of a running VM. you must
> stop it before.  

Ok, so it is a matter of running 24/7 - but when do so,  you usually
have multiple redundancies, so may be placing one VM into one dataset
could help to avoid copies - as this way you could switch one off while
backing it up without hampering the whole shebang.

Note that this would also help to keep more snapshots, just in case.

There's one thing I do not understand, you talk about consolidating
snapshots, but you usually just discard them when not needed anymore.

Jean-Yves

From tschoening at am-soft.de  Sat Sep  4 05:20:02 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Sat, 4 Sep 2021 11:20:02 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
Message-ID: <114179103.20210904112002@am-soft.de>

Guten Tag Ernest CHIARELLO,
am Freitag, 3. September 2021 um 15:32 schrieben Sie:

> I want to clarify my request.[...]

No, you are changing it and I storngly suggest that further discussion
should be made in a new, indiviodual topic. Remember that the former
topic was about why BorgBackup considers your unchanged files changed.
You know the reason now and a workaround.

What you are describing/answering for additionally is more about a
suggestion about your overall backup use-case. That is a totally
different topic, otherwise you wouldn't need to describe it. :-)

> my question is: would it be possible to replace the "cp -u" by
> "borg create" (from /var/lib/libvirt/images to
> /var/lib/libvirt/backups), in order to benefit from the
> deduplication and to shorten the copy times ?

No, because you are mixing two different topics. If you want to backup
the individual files created by "fi-backup", then of course you can
use Borg to do so. Simply because Borg doesn't care too much about the
purpose of the files you are backing up.

But is that logically the same as letting "fi-backup" copy its own
files, create larger images and back those up? No. It's different. Is
it what you need/want? Don't know, that's something only you can
decide on your own.

Anyway, the result of using "borg create" in different steps won't
ever be the same as some "cp -u", because on one case you have content
added to a Borg repo always, while in the other case you have a
completely different file format maintained by "fi-backup". Most
likely QCOW2 if I understand the docs correctly.

> the creation of dummy files would then also be managed by fi-backup.

If those are necessary at all, in my opinion those dummy files a an
imlpementation detail for Borg and should therefore be managed by Borg
or, as suggested, BorgMatic instead. "fi-backup" doesn't know or need
to care about what Borg consideres changed or unchanged, when it does
read which files again for wehatever reason etc.

> i suppose you can not save the qcow2 file of a running VM. you must
> stop it before.

That totally depends on the content of the image. A lot of operating
systems, file systems and even apps like databases or else can handle
and survive a crashing host pretty well. just think about your own
Windows/Linux computer you use for your daily work or privately, they
crashed most likely at some point in the past. With whcih results,
completely data loss, new hardware necessary? Most likely not, only
some unsaved changes might have been lost, some FSCK/CHECKDISK being
executed and afterwards you were good to go again.

That's not much different with your VMs most likely and allows you to
backup running VMs as well. The important thing is to freeze the
running VM for how long the backup takes and you have multiple ways to
do so: Suspending the VM or something like file system level snapshots
of e.g. BTRFS and ZFS.

In my opinion, most people these days prefer the latter: Putting VMs
into BTRFS/ZFS, create a snapshot, mount that snapshots and let
BorgBackup directly backup that snapshot. This doesn't interrupt the
oepration of the VM, while at the same time what you get is a so
called "crash consistent" backup. That is enough for many needs, but
of course you need to decide that on your own.

And here's the point: When doing that approach, while BorgBackup reads
the whole image, in that use-case it directly applies de-duplication,
compression etc. and really only writes the changed data to its own
repo. If that is faster or not than what "fi-backup" does in two steps
is something you need to test on your own. Though, without knowing
"fi-backup", in my opinion having only one step is easier in the end.
Remember that each run of "borg create" will provide you a working VM,
without additional steps to copy things around, merge QCOW2 images
etc.

Additionally have a look at what "fi-backup" writes on its own:

> By default fi-backup.sh guarantees that qcow2 images format is
> consistent, but this doesn't imply that the contained filesystems
> are consistent too. In order to perform consistent backups, you can
> use two different strategies:

https://github.com/dguerri/LibVirtKvm-scripts#note

So, depending on your setup, you might already backup only "crash
consistent" state anyway. Additionally, even with the guest agend of
QEMU installed, individual apps like databases won't know anything
about that "fi-backup" is doing some backup "right now" and therefore
won't do anything to make their own files consistent. The only thing
the agent changes is that "fi-backup" is able to tell the guest-VM to
e.g. sync pending file system changes on disk before suspending. that
DOES NOT guarantee any consistent file format state in the VM, though.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From tschoening at am-soft.de  Mon Sep  6 13:19:12 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Mon, 6 Sep 2021 19:19:12 +0200
Subject: [Borgbackup] Can different error codes of individual files be
 mapped to different results?
Message-ID: <1181662031.20210906191912@am-soft.de>

Hi everyone,

I'm backing things up using SSHFS and ran into two different, but
as well related problems:

# Dial-up DSL

Sometimes SSHFS is hosted by some dial-up DSL internet connection and
that connection might simply reconnect during the time a backup is
executed. SSHFS simply disappears in those cases. When this happens,
it seems that BORG only recognizes missing files and considers that a
warning instead of an error.

> [2021-09-03 01:53:32,906] INFO: /mnt/[...]/root_and_some_zfs/root/vmlinuz.old: stat: [Errno 2] No such file or directory: '/mnt/[...]/root_and_some_zfs/root/vmlinuz.old'
> [2021-09-03 01:53:32,906] INFO: E /mnt/[...]/root_and_some_zfs/root/vmlinuz.old
> [...]
> [2021-09-03 01:53:33,575] INFO: security: saving state for [...] to /home/ams_d_bak_borg/.config/borg/security/[...]
> [2021-09-03 01:53:34,918] INFO: security: current location   ssh://[...]
> [2021-09-03 01:53:34,918] INFO: security: key type           0
> [2021-09-03 01:53:34,918] INFO: security: manifest timestamp 2021-09-02T23:53:32.947038
> [2021-09-03 01:53:34,919] INFO: RemoteRepository: 3.22 MB bytes sent, 3.21 kB bytes received, 116 messages sent
> [2021-09-03 01:53:34,948] INFO: terminating with warning status, rc 1
> [2021-09-03 01:53:34,984] INFO: [...]: Running consistency checks

# Expected missing files

I'm backing up using SSHFS from different devices, like some NAS. That
NAS stores backups from some Linux hosts and those backups contain
symlinks using absolute paths as well. Those symlinks can't be
properly resolved on the NAS of course, so it answers with missing
files. Compared to the former problem this time missing files are OK,
because I know things can't work differently and therefore BORG
aborting with errors wouldn't make too much sense.

> [2021-08-25 09:17:45,430] INFO: /mnt/[...]/root/cdrom: readlink: [Errno 2] No such file or directory: '/mnt/[...]/root/cdrom'
> [2021-08-25 09:17:45,430] INFO: E /mnt/[...]/root/cdrom

# BORGMATIC

I'm using BORGMATIC for individual backups, which allows me to easily
maintain individual settings for BORG and forward additional options.
If BORG would support some kind of switch to influence when missing
files are an error and when not, I would easiyl be able to distinguish
both use-cases above. In most cases I won't have any missing files.

# Error code mapping?

So, is there some error code mapping of individual files or alike?
Something to tell BORG when to exit with a warning compared to an
error? Is there any option I can change what BORG considers to be an
error under whcih circumstances?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From tw at waldmann-edv.de  Mon Sep  6 15:54:40 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Mon, 6 Sep 2021 21:54:40 +0200
Subject: [Borgbackup] Can different error codes of individual files be
 mapped to different results?
In-Reply-To: <1181662031.20210906191912@am-soft.de>
References: <1181662031.20210906191912@am-soft.de>
Message-ID: <7e58fc55-f1b3-ec52-03d5-d9365ff57161@waldmann-edv.de>


> Sometimes SSHFS is hosted by some dial-up DSL internet connection and
> that connection might simply reconnect during the time a backup is
> executed. SSHFS simply disappears in those cases.

This is a bug in sshfs, please use their issue tracker to report and 
discuss this.

Expected behaviour of a mounted, but non-working filesystem is to yield 
errors, not just appear empty.

Thus, no borg issue here.

> When this happens,
> it seems that BORG only recognizes missing files and considers that a
> warning instead of an error.

Borg has no expectation about how your files / directories should look like.

It just relies on whatever you give to it being the state which you 
actually want to back up.

If you give it an empty directory, it will be happy to backup that for you.

Also, if a filename was present in a directory, but later (when trying 
to open and read it) is gone, borg will just log a warning.

Warning means that borg has no strong opinion about whether this is OK 
or an ERROR, but that this needs to be reviewed by the operator. 
Frequently happens for temporary files or otherwise active file systems.

Thus, no borg issue here.

> Those symlinks can't be properly resolved on the NAS of course,
> so it answers with missing files.

When accessing symlinks, there are 2 modes: follow and nofollow.

When backing up, borg usually uses NOFOLLOW, so it just backs up the 
symlink, not the object it points to.

There is some exception for --read-special, see the docs.

So, no borg issue here. Maybe your network filesystem / NAS is doing 
something wrong?

You can also just try this on a local filesystem (like ext4) to verify 
borg's behaviour.

> So, is there some error code mapping of individual files or alike? 

Problems with "disappearing" files are usually due to backing up temp 
files (solution: just exclude temp locations) or due to active 
filesystem (solution: do a snapshot, so it is not actively changing).

Alternatively, review the warnings manually and ignore the ones which 
are no problem.



From tschoening at am-soft.de  Tue Sep  7 02:24:59 2021
From: tschoening at am-soft.de (=?windows-1250?Q?Thorsten_Sch=F6ning?=)
Date: Tue, 7 Sep 2021 08:24:59 +0200
Subject: [Borgbackup] Can different error codes of individual files be
 mapped to different results?
In-Reply-To: <7e58fc55-f1b3-ec52-03d5-d9365ff57161@waldmann-edv.de>
References: <1181662031.20210906191912@am-soft.de>
 <7e58fc55-f1b3-ec52-03d5-d9365ff57161@waldmann-edv.de>
Message-ID: <148049525.20210907082459@am-soft.de>

Guten Tag Thomas Waldmann,
am Montag, 6. September 2021 um 21:54 schrieben Sie:

> Thus, no borg issue here.

I didn't claim any Borg issue at all, only wanted to describe my
use-cases and see if I missed something. Possibly starting a
discussion about some in my opinion useful enhancement.

> Alternatively, review the warnings manually and ignore the ones
> which are no problem.

That doesn't help anything in context of e.g. using BorgMatic, as that
relies on the overall result code of Borg to decide which hook to call
after the backup. Additionally, manually looking at logs with
thousands of files is unnecessary error-prone and time consuming as
well.

In theory, Borg would be able to implement two different strategies
making use-cases like mine easier to handle: First, keep track of all
warnings/errors per file using similar approaches like the file cache.
At the end, those would be mapped to some overall result according to
some option given. The benefit is that things could be summarized or
else at the end as well to provide further details of Borg's decision.

The second approach is to simply decide for each first occuring
warning/error if it should be mapped to some overall result already
and store that. Means far less data to maintain and one might not care
about each and every individual file with that warning or error at
all. In my case I e.g. have full debug logs containing all details
anyway, which are at least part of later backups as well.

In both cases its only about overall result codes, not aborting
earlier than before. Borg should simply continue to try to backup what
it is able to read.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From Ernest.Chiarello at univ-fcomte.fr  Tue Sep  7 04:34:14 2021
From: Ernest.Chiarello at univ-fcomte.fr (Chiarello Ernest)
Date: Tue, 7 Sep 2021 10:34:14 +0200
Subject: [Borgbackup] tag A or U when files are updated ?
In-Reply-To: <114179103.20210904112002@am-soft.de>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
 <114179103.20210904112002@am-soft.de>
Message-ID: <5339e7c7-8e27-6014-cc1b-4a224cd2af4e@univ-fcomte.fr>

dear Thorsten,

you are absolutely right ...

my apologies for this change of subject.

this subject is therefore closed. A big thank you for your reply. ?

bonne journ?e,

Ernest.

Le 04/09/2021 ? 11:20, Thorsten Sch?ning a ?crit?:
> Guten Tag Ernest CHIARELLO,
> am Freitag, 3. September 2021 um 15:32 schrieben Sie:
>
>> I want to clarify my request.[...]
> No, you are changing it and I storngly suggest that further discussion
> should be made in a new, indiviodual topic. Remember that the former
> topic was about why BorgBackup considers your unchanged files changed.
> You know the reason now and a workaround.



-- 
Ernest CHIARELLO - Ernest.Chiarello at univ-fcomte.fr

UMR6049 Th?MA - CNRS / Universit? de Franche-Comt?
Service de gestion et production de l'information
32 rue M?gevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

USR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210907/89e9c879/attachment-0001.html>

From Ernest.Chiarello at univ-fcomte.fr  Tue Sep  7 05:54:30 2021
From: Ernest.Chiarello at univ-fcomte.fr (Chiarello Ernest)
Date: Tue, 7 Sep 2021 11:54:30 +0200
Subject: [Borgbackup] use of borgbackup in fi-backup to save VM images
In-Reply-To: <114179103.20210904112002@am-soft.de>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
 <114179103.20210904112002@am-soft.de>
Message-ID: <621ef90d-d30a-871a-3e80-20eeb08bf4f7@univ-fcomte.fr>

bonjour Thorsten, hi everyone ;

Le 04/09/2021 ? 11:20, Thorsten Sch?ning a ?crit?:
> Guten Tag Ernest CHIARELLO,
> am Freitag, 3. September 2021 um 15:32 schrieben Sie:
>
>> my question is: would it be possible to replace the "cp -u" by
>> "borg create" (from /var/lib/libvirt/images to
>> /var/lib/libvirt/backups), in order to benefit from the
>> deduplication and to shorten the copy times ?
> No, because you are mixing two different topics. If you want to backup
> the individual files created by "fi-backup", then of course you can
> use Borg to do so. Simply because Borg doesn't care too much about the
> purpose of the files you are backing up.
>
> But is that logically the same as letting "fi-backup" copy its own
> files, create larger images and back those up? No. It's different. Is
> it what you need/want? Don't know, that's something only you can
> decide on your own.
>
> Anyway, the result of using "borg create" in different steps won't
> ever be the same as some "cp -u", because on one case you have content
> added to a Borg repo always, while in the other case you have a
> completely different file format maintained by "fi-backup". Most
> likely QCOW2 if I understand the docs correctly.
>
>> the creation of dummy files would then also be managed by fi-backup.
> If those are necessary at all, in my opinion those dummy files a an
> imlpementation detail for Borg and should therefore be managed by Borg
> or, as suggested, BorgMatic instead. "fi-backup" doesn't know or need
> to care about what Borg consideres changed or unchanged, when it does
> read which files again for wehatever reason etc.
OK
>> i suppose you can not save the qcow2 file of a running VM. you must
>> stop it before.
> That totally depends on the content of the image. A lot of operating
> systems, file systems and even apps like databases or else can handle
> and survive a crashing host pretty well. just think about your own
> Windows/Linux computer you use for your daily work or privately, they
> crashed most likely at some point in the past. With whcih results,
> completely data loss, new hardware necessary? Most likely not, only
> some unsaved changes might have been lost, some FSCK/CHECKDISK being
> executed and afterwards you were good to go again.
>
> That's not much different with your VMs most likely and allows you to
> backup running VMs as well. The important thing is to freeze the
> running VM for how long the backup takes and you have multiple ways to
> do so: Suspending the VM or something like file system level snapshots
> of e.g. BTRFS and ZFS.

i know zfs is used by ProxMox, as you can read here : ZFS on proxmox 
<https://pve.proxmox.com/wiki/ZFS_on_Linux>

But btrfs is not recommended according 
tohttp://www.linux-kvm.org/page/Tuning_KVM 
<http://www.linux-kvm.org/page/Tuning_KVM>

/Don't use the linux filesystem btrfs on the host for the image files. 
It will result in low IO performance. The kvm guest may even freeze when 
high IO traffic is done on the guest. /

> In my opinion, most people these days prefer the latter: Putting VMs
> into BTRFS/ZFS, create a snapshot, mount that snapshots and let
> BorgBackup directly backup that snapshot. This doesn't interrupt the
> oepration of the VM, while at the same time what you get is a so
> called "crash consistent" backup. That is enough for many needs, but
> of course you need to decide that on your own.
>
> And here's the point: When doing that approach, while BorgBackup reads
> the whole image, in that use-case it directly applies de-duplication,
> compression etc. and really only writes the changed data to its own
> repo. If that is faster or not than what "fi-backup" does in two steps
> is something you need to test on your own. Though, without knowing
> "fi-backup", in my opinion having only one step is easier in the end.
> Remember that each run of "borg create" will provide you a working VM,
> without additional steps to copy things around, merge QCOW2 images
> etc.
thank you for this excellent understanding of the situation and for your 
good advice! ?
> Additionally have a look at what "fi-backup" writes on its own:
>
>> By default fi-backup.sh guarantees that qcow2 images format is
>> consistent, but this doesn't imply that the contained filesystems
>> are consistent too. In order to perform consistent backups, you can
>> use two different strategies:
> https://github.com/dguerri/LibVirtKvm-scripts#note
>
> So, depending on your setup, you might already backup only "crash
> consistent" state anyway. Additionally, even with the guest agend of
> QEMU installed, individual apps like databases won't know anything
> about that "fi-backup" is doing some backup "right now" and therefore
> won't do anything to make their own files consistent. The only thing
> the agent changes is that "fi-backup" is able to tell the guest-VM to
> e.g. sync pending file system changes on disk before suspending. that
> DOES NOT guarantee any consistent file format state in the VM, though.

you are quite right to recall that.

this is the reason why I use Urbackup <https://www.urbackup.org/> to 
back up data contained in VMs (files and databases).

thank you for this interesting discussion,

bonne journ?e,

Ernest.

> Mit freundlichen Gr??en
>
> Thorsten Sch?ning
>

-- 
Ernest CHIARELLO -Ernest.Chiarello at univ-fcomte.fr

UMR6049 Th?MA - CNRS / Universit? de Franche-Comt?
Service de gestion et production de l'information
32 rue M?gevand - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08

USR3124 MSHE Ledoux -  CNRS / UFC
Plateforme technologique SHERPA
1 rue Charles Nodier - F 25030 BESANCON Cedex
Tel : 03 81 66 53 66           Mob : 07 82 99 11 08


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210907/34f741db/attachment.html>

From tschoening at am-soft.de  Tue Sep  7 06:47:53 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Tue, 7 Sep 2021 12:47:53 +0200
Subject: [Borgbackup] use of borgbackup in fi-backup to save VM images
In-Reply-To: <621ef90d-d30a-871a-3e80-20eeb08bf4f7@univ-fcomte.fr>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
 <114179103.20210904112002@am-soft.de>
 <621ef90d-d30a-871a-3e80-20eeb08bf4f7@univ-fcomte.fr>
Message-ID: <696906557.20210907124753@am-soft.de>

Guten Tag Chiarello Ernest,
am Dienstag, 7. September 2021 um 11:54 schrieben Sie:

> /Don't use the linux filesystem btrfs on the host for the image
> files. It will result in low IO performance. The kvm guest may even
> freeze when high IO traffic is done on the guest. /

Some of those tests/advices are old, so it might be worth it to test
on your own. While I prefer ZFS in general, things like freezing VMs
are often seen when hosts have a lot of free RAM for file system cache
available. I had exactly that with VBOX and ZFS in the past, because
the virtual controllers used for IDE/SATA etc. of VBOX were configured
to cache. For some use-cases this resulted in so many pending writes
in the cache, that the HDDs backing ZFS weren't able to catch up with.

Though, after disabling those caches things were fine again. And
Proxmox provides ways to disable those caches regardless of the
underlying file system as well, so things might simply not be as bad
as you quoted for BTRFS.

https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache

OTOH, as Proxmox seems to recommend ZFS anyway and supports that in
its own installers etc., I would simply use that instead of BTRFS.

> this is the reason why I use Urbackup <https://www.urbackup.org/>
> to back up data contained in VMs (files and databases).

Remember that only because you backup VM-contents that doesn't
guarantee any consistent individual file format state as well. Things
are most likely only SYNC-safe and hence crash consistent as well, you
need to make snapshots within the VMs in theory as well etc.

UrBackup might handle all of that for all supported OS or it might
not, better look twice. Especially as Linux lacks infrastructure like
VSS under Windows telling apps about snapshots beeing created, to make
their individual file formats consistent etc. It might be that
UrBackup only cares on Windows because VSS is available by default.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From devzero at web.de  Tue Sep  7 12:01:01 2021
From: devzero at web.de (devzero at web.de)
Date: Tue, 7 Sep 2021 18:01:01 +0200
Subject: [Borgbackup] use of borgbackup in fi-backup to save VM images
In-Reply-To: <696906557.20210907124753@am-soft.de>
References: <8a740013-3273-11c8-6196-8a8ab797fd6f@univ-fcomte.fr>
 <647998900.20210902130105@am-soft.de>
 <e857875c-b706-58fd-e81d-44ef43f09320@univ-fcomte.fr>
 <14910591148.20210902154848@am-soft.de>
 <85dfab9d-0784-656e-e163-8ccdfd373fdd@univ-fcomte.fr>
 <114179103.20210904112002@am-soft.de>
 <621ef90d-d30a-871a-3e80-20eeb08bf4f7@univ-fcomte.fr>
 <696906557.20210907124753@am-soft.de>
Message-ID: <trinity-f9ffc6dd-cd56-45b5-8597-f93a4592deb9-1631030461611@3c-app-webde-bap09>

> Though, after disabling those caches things were fine again. 

*sigh*

there are longstanding bugticket regarding freezing, see  https://bugzilla.proxmox.com/show_bug.cgi?id=1453 and https://bugzilla.kernel.org/show_bug.cgi?id=199727 for example

think it needs some kvm & zfs expert to get this resolved... 

at least proxmox team is willing to investigate into this. apparently, kvm folks completely ignore bug #199727 since 2018...

regards
roland


> Gesendet: Dienstag, 07. September 2021 um 12:47 Uhr
> Von: "Thorsten Sch?ning" <tschoening at am-soft.de>
> An: Borgbackup at python.org
> Betreff: Re: [Borgbackup] use of borgbackup in fi-backup to save VM images
>
> Guten Tag Chiarello Ernest,
> am Dienstag, 7. September 2021 um 11:54 schrieben Sie:
> 
> > /Don't use the linux filesystem btrfs on the host for the image
> > files. It will result in low IO performance. The kvm guest may even
> > freeze when high IO traffic is done on the guest. /
> 
> Some of those tests/advices are old, so it might be worth it to test
> on your own. While I prefer ZFS in general, things like freezing VMs
> are often seen when hosts have a lot of free RAM for file system cache
> available. I had exactly that with VBOX and ZFS in the past, because
> the virtual controllers used for IDE/SATA etc. of VBOX were configured
> to cache. For some use-cases this resulted in so many pending writes
> in the cache, that the HDDs backing ZFS weren't able to catch up with.
> 
> Though, after disabling those caches things were fine again. And
> Proxmox provides ways to disable those caches regardless of the
> underlying file system as well, so things might simply not be as bad
> as you quoted for BTRFS.
> 
> https://pve.proxmox.com/wiki/Performance_Tweaks#Disk_Cache
> 
> OTOH, as Proxmox seems to recommend ZFS anyway and supports that in
> its own installers etc., I would simply use that instead of BTRFS.
> 
> > this is the reason why I use Urbackup <https://www.urbackup.org/>
> > to back up data contained in VMs (files and databases).
> 
> Remember that only because you backup VM-contents that doesn't
> guarantee any consistent individual file format state as well. Things
> are most likely only SYNC-safe and hence crash consistent as well, you
> need to make snapshots within the VMs in theory as well etc.
> 
> UrBackup might handle all of that for all supported OS or it might
> not, better look twice. Especially as Linux lacks infrastructure like
> VSS under Windows telling apps about snapshots beeing created, to make
> their individual file formats consistent etc. It might be that
> UrBackup only cares on Windows because VSS is available by default.
> 
> Mit freundlichen Gr??en
> 
> Thorsten Sch?ning
> 
> -- 
> AM-SoFT IT-Service - Bitstore Hameln GmbH
> Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK
> 
> E-Mail: Thorsten.Schoening at AM-SoFT.de
> Web:    http://www.AM-SoFT.de/
> 
> Tel:   05151-  9468- 0
> Tel:   05151-  9468-55
> Fax:   05151-  9468-88
> Mobil:  0178-8 9468-04
> 
> AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
> AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska
> 
> 
> 
> 
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup
>

From tw at waldmann-edv.de  Wed Sep  8 06:33:26 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Wed, 8 Sep 2021 12:33:26 +0200
Subject: [Borgbackup] Can different error codes of individual files be
 mapped to different results?
In-Reply-To: <148049525.20210907082459@am-soft.de>
References: <1181662031.20210906191912@am-soft.de>
 <7e58fc55-f1b3-ec52-03d5-d9365ff57161@waldmann-edv.de>
 <148049525.20210907082459@am-soft.de>
Message-ID: <5615477f-fa2d-847a-ebac-e16874093720@waldmann-edv.de>


>> Alternatively, review the warnings manually and ignore the ones
>> which are no problem.
> That doesn't help anything in context of e.g. using BorgMatic,
Well, then use the other alternatives I gave.
> In theory, Borg would be able to implement two different strategies
> making use-cases like mine easier to handle: First, keep track of all
> warnings/errors per file using similar approaches like the file cache.
> At the end, those would be mapped to some overall result according to
> some option given. The benefit is that things could be summarized or
> else at the end as well to provide further details of Borg's decision.
>
> The second approach is to simply decide for each first occuring
> warning/error if it should be mapped to some overall result already
> and store that. Means far less data to maintain and one might not care
> about each and every individual file with that warning or error at
> all. In my case I e.g. have full debug logs containing all details
> anyway, which are at least part of later backups as well.

You still would need borg to decide whether some warning is severe or not.

It can't do that automagically and configuring that per-file would be 
way too tedious.


From tschoening at am-soft.de  Wed Sep  8 09:15:45 2021
From: tschoening at am-soft.de (=?windows-1250?Q?Thorsten_Sch=F6ning?=)
Date: Wed, 8 Sep 2021 15:15:45 +0200
Subject: [Borgbackup] Can different error codes of individual files be
 mapped to different results?
In-Reply-To: <5615477f-fa2d-847a-ebac-e16874093720@waldmann-edv.de>
References: <1181662031.20210906191912@am-soft.de>
 <7e58fc55-f1b3-ec52-03d5-d9365ff57161@waldmann-edv.de>
 <148049525.20210907082459@am-soft.de>
 <5615477f-fa2d-847a-ebac-e16874093720@waldmann-edv.de>
Message-ID: <1512247710.20210908151545@am-soft.de>

Guten Tag Thomas Waldmann,
am Mittwoch, 8. September 2021 um 12:33 schrieben Sie:

>> That doesn't help anything in context of e.g. using BorgMatic,
> Well, then use the other alternatives I gave.

Your arguments are mostly missing the point, it's neither about
temporary directories, snapshots are already used etc. It's only about
whoever is interested in BORG's overall result code.

The only alternatives are lettings others look into BORG's logs, but
that requires one to log those files and their results, one needs to
parse possibly large text files etc. Would be easier and more
efficient if BORG could handle that on its won already.

https://projects.torsion.org/witten/borgmatic/issues/447

> It can't do that automagically and configuring that per-file would
> be way too tedious.

BORG doesn't need and even shouldn't decide automagically. We only
need to find an easy way to map errors of individual files to the one
and only overall result code.

In the easiest case one doesn't need configs per file, but is only
interested in if any error (like missing file) happened at all. So
whenever BORG detects an error during backup of some file, it decides
based on that error code if the overall result might be an error as
well. The error code per file is available aready, as it is logged and
mapped to letter "E". So an additional switch to provide error codes
for individual files to be handled as error instead of a warning might
be enough already.

> borg create --file-result-is-error always|2|5|...

In the long term, this can be expanded with file paths, multiple
provided error codes etc. using regular expressions. Simply like
"--exclude", "--exclude-from" etc. work already.

> borg create --file-result-is-error-from

The underlying file format could e.g. simply be some space separated
map of result codes mapped to paths. This file would answer the
following questions: If the error code of the current file is "..."
and if the current file matches "...", consider the overall result to
be an error.

> 2 /mnt/.+/root/vmlinuz.old
> 5 /mnt/.+/root/some_file.txt

With an additional column this could even be enhanced further to be a
concrete overall result code like 25 instead of the default 2, it
could be forcefully mapped to be a warning with result code 1 only
etc. This would only be needed for files with errors, which in theory
are less, and would be much faster and safer than parsing logs.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From tw at waldmann-edv.de  Wed Sep  8 09:34:15 2021
From: tw at waldmann-edv.de (Thomas Waldmann)
Date: Wed, 8 Sep 2021 15:34:15 +0200
Subject: [Borgbackup] Can different error codes of individual files be
 mapped to different results?
In-Reply-To: <1512247710.20210908151545@am-soft.de>
References: <1181662031.20210906191912@am-soft.de>
 <7e58fc55-f1b3-ec52-03d5-d9365ff57161@waldmann-edv.de>
 <148049525.20210907082459@am-soft.de>
 <5615477f-fa2d-847a-ebac-e16874093720@waldmann-edv.de>
 <1512247710.20210908151545@am-soft.de>
Message-ID: <91407969-b6ff-1c04-3210-bd70fa3de983@waldmann-edv.de>

I disagree and I won't implement any tedious returncode tweaking mapping 
mechanism to solve:

- stuff that can be done by a borg wrapper or borg log analyzer

- bugs in other software, like sshfs suddenly getting empty instead of 
yielding errors (which seems to be your root problem)


borg's return codes are simply:

- 0 (all OK)

- 1 (warning - you have to look yourself whether they are serious or not 
[often it is an issue with one or a few files], the backup reached its 
normal end)

- 2 (error - usually some sudden fatal error and the backup did not 
complete to its end)


if you're unhappy with that, write a borg wrapper.



From tschoening at am-soft.de  Tue Sep 14 04:54:11 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Tue, 14 Sep 2021 10:54:11 +0200
Subject: [Borgbackup] Experience with compressed database dumps?
Message-ID: <1795172926.20210914105411@am-soft.de>

Hi everyone,

I have some BORG-repos storing database dumps of MySQL and Postgres
only. The current approach is that CRON dumps databases into files and
the used scripts compress those dumps as well already for historical
reasons. This is either done by using BZIP2 for a SQL-based dump or in
case fof Postgres and pg_dump using its own compressed file format.

Reason is that those dumps were backed up to some uncompressed storage
in the past, then BTRFS was used and it didn't make a difference too
much if that or the backup script compressed already. But with BORG
de-duplication comes into play.

I wonder if it's better to store uncompressed data and let BORG handle
de-duplication and compression or if it doesn't matter too much
anyway? Does anyone hves experiences/numbers/... already for that
use-case?

Thanks!

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From oh at dom.de  Fri Sep 17 12:14:17 2021
From: oh at dom.de (Oliver Hoffmann)
Date: Fri, 17 Sep 2021 18:14:17 +0200
Subject: [Borgbackup] Moving from old to new backup server
Message-ID: <c0856ff2-838b-8527-a4f0-a21028723e9e@dom.de>

Hi list,


I need to move my borg server to a new machine due to capacity issues.

So, the idea is to simply rsync all the data, which is about 25 TB to
the new one and then migrating the VM as well. No backups while moving
and a final rsync of course.

borg is running on a VM (Debian on xcp-ng). On the dom0 runs a nfs
server; the VM is nfs client. In the end borg puts all the repositories
in /mnt/nfs. Sounds a bit weird but it works fine and that way it is
more flexible. Hopefully just now too ;-)

Otherwise I could set up an "empty" borg-VM and move all the clients to
the new one. After a couple of weeks when the backups are old and
obsolete, I could switch off the old borg server.

I like the other idea more though. What do you think?

Oh, another question. What would be the best file system? Just ext4 or
better something fancier like btfs or xfs? zfs would be nonsense as the
raid is handled by hardware.
I have at least 50 clients (big and small files) and the raid is about
36 TB.

Thank you for your input!

Regards,

Oliver

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210917/8189283e/attachment.sig>

From fabio.pedretti at unibs.it  Fri Sep 17 12:25:51 2021
From: fabio.pedretti at unibs.it (Fabio Pedretti)
Date: Fri, 17 Sep 2021 18:25:51 +0200
Subject: [Borgbackup] Experience with compressed database dumps?
In-Reply-To: <1795172926.20210914105411@am-soft.de>
References: <1795172926.20210914105411@am-soft.de>
Message-ID: <CAGpfsQ0nU1=WE_srF-pNnnP8Cn579byrCLx7bwAD5f_8SAy9mg@mail.gmail.com>

The best compromise I was able to use for my use case is to compress
with "zstd --rsyncable", which preserve deduplication.
Regards.

Il giorno mar 14 set 2021 alle ore 10:54 Thorsten Sch?ning
<tschoening at am-soft.de> ha scritto:
>
> Hi everyone,
>
> I have some BORG-repos storing database dumps of MySQL and Postgres
> only. The current approach is that CRON dumps databases into files and
> the used scripts compress those dumps as well already for historical
> reasons. This is either done by using BZIP2 for a SQL-based dump or in
> case fof Postgres and pg_dump using its own compressed file format.
>
> Reason is that those dumps were backed up to some uncompressed storage
> in the past, then BTRFS was used and it didn't make a difference too
> much if that or the backup script compressed already. But with BORG
> de-duplication comes into play.
>
> I wonder if it's better to store uncompressed data and let BORG handle
> de-duplication and compression or if it doesn't matter too much
> anyway? Does anyone hves experiences/numbers/... already for that
> use-case?
>
> Thanks!
>
> Mit freundlichen Gr??en
>
> Thorsten Sch?ning
>
> --
> AM-SoFT IT-Service - Bitstore Hameln GmbH
> Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK
>
> E-Mail: Thorsten.Schoening at AM-SoFT.de
> Web:    http://www.AM-SoFT.de/
>
> Tel:   05151-  9468- 0
> Tel:   05151-  9468-55
> Fax:   05151-  9468-88
> Mobil:  0178-8 9468-04
>
> AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
> AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska
>
>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup

-- 


Informativa sulla Privacy: https://www.unibs.it/it/node/1452 
<https://www.unibs.it/it/node/1452>

From tschoening at am-soft.de  Sat Sep 18 05:44:31 2021
From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=)
Date: Sat, 18 Sep 2021 11:44:31 +0200
Subject: [Borgbackup] Moving from old to new backup server
In-Reply-To: <c0856ff2-838b-8527-a4f0-a21028723e9e@dom.de>
References: <c0856ff2-838b-8527-a4f0-a21028723e9e@dom.de>
Message-ID: <999439927.20210918114431@am-soft.de>

Guten Tag Oliver Hoffmann,
am Freitag, 17. September 2021 um 18:14 schrieben Sie:

> Otherwise I could set up an "empty" borg-VM and move all the clients to
> the new one. After a couple of weeks when the backups are old and
> obsolete, I could switch off the old borg server.

BorgMatic might have an interesting different idea as well: Keep the
setup as-is and start to backup into multiple repos, some hosted on
your new server. If your old system really can't be used anymore or
the backups on the new system are simply enough, just switch your VM.

https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/

This would prevent RSYNC-times, creating a new VM and your restore
etc. could be kept as-is as long as storage is available on the old
server. OTOH, backup times increase from my understanding.

> I like the other idea more though. What do you think?

The main point might be if RSYNC is able to catch up with changes,
like when pruning archives. From my understanding, compacting results
in new additional files, which might make RSYNC transfer files today
which are ocmpacted and removed tomorrow in favour of additionally to
transfer files. If that happens more often than RSYNC is able to
transfer data, your migration might never finish at all in theory. :-)

Though, I guess that's pretty unlikely. If I was you, I would decide
based on how wortht he old backups in the new systems are and how long
backing up to two destinations might take. Because it might be of
benefit to really start over with new repos with new default settings
and stuff.

Otherwise, your RSYNC-approach reads fine to me and I would most
likely do the same.

> Oh, another question. What would be the best file system? Just ext4 or
> better something fancier like btfs or xfs? zfs would be nonsense as the
> raid is handled by hardware.

ZFS is not just about RAID, just as BTRFS is not only about non-RAID.
In fact, ZFS seems abit more mature, e.g. cares about things like file
name encoding instead of simply storing bytes and stuff like that. In
my opinion, it has the better interface and some better concepts like
pools vs. datasets vs. snapshots vs. clones etc. as well. There might
even be better tooling like zfs-auto-snaps, ZREP and stuff like that.
Depending on where you run your hardware, built-in encryption might be
a good argument as well, it is for me.

Said that, have a look at your distribution first and which file
system it prefers and is best integrated with. Think of updates and
problems during those, some distributions integrate tightly with
BTRFS, like SUSE, some chose ZFS instead, like Ubuntu, and provide a
lot of default tools and setup for either of both. Not only because of
updates, but backup as well.

For me, it's always BTRFS or ZFS these days, depending on the used OS,
with ZFS being in favor. XFS is in the process of catching up only and
lacks a lot of tooling and features, ext4 lacks essential concepts
like snapshots, compression etc. and seems pretty legacy to me. No
reason to start new systems with the last two mentioned in my opinion.

Mit freundlichen Gr??en

Thorsten Sch?ning

-- 
AM-SoFT IT-Service - Bitstore Hameln GmbH
Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK

E-Mail: Thorsten.Schoening at AM-SoFT.de
Web:    http://www.AM-SoFT.de/

Tel:   05151-  9468- 0
Tel:   05151-  9468-55
Fax:   05151-  9468-88
Mobil:  0178-8 9468-04

AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln
AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska





From liori at exroot.org  Sat Sep 18 06:15:14 2021
From: liori at exroot.org (Tomasz Melcer)
Date: Sat, 18 Sep 2021 12:15:14 +0200
Subject: [Borgbackup] Experience with compressed database dumps?
In-Reply-To: <1795172926.20210914105411@am-soft.de>
References: <1795172926.20210914105411@am-soft.de>
Message-ID: <6ea88a23-c047-16f9-7479-cc6737c7c4d3@exroot.org>

On 14.09.2021 10:54, Thorsten Sch?ning wrote:
> I wonder if it's better to store uncompressed data and let BORG handle
> de-duplication and compression or if it doesn't matter too much
> anyway? Does anyone hves experiences/numbers/... already for that
> use-case?

I'm doing mostly what you describe for a SQL Server database in a 
process I'm using for periodic migration of production data to a staging 
database. I suspect it will be similar to PostgreSQL/MySQL. My workflow 
is: dump databases using `bcp` into btrfs (compressed), then run 
borgbackup over it. Roughly: ~40k tables, 700 GB of database files ? 300 
GB of raw dumped data ? 100 GB of btrfs-compressed data ? 500 MB to 5 GB 
of borgbackup lzma-compressed increments. Some actual numbers from the 
last run as reported by `borg info`:

Duration: 9 hours 15 minutes 55.04 seconds

(Original size / Compressed size / Deduplicated size)

This archive: 362.90 GB / 5.65 GB / 1.82 GB

All archives: 4.57 TB / 69.77 GB / 18.47 GB

Definitely good enough for my purposes, though as you can see, it takes 
several hours for borg to process this data. For comparison: just 
dumping the data with `bcp` takes ~2.5h using a 200Mbps connection. In 
my case the procedure is not time-sensitive, so it's fine.


-- 
Tomasz Melcer

From oh at dom.de  Sun Sep 19 15:42:59 2021
From: oh at dom.de (Oliver Hoffmann)
Date: Sun, 19 Sep 2021 21:42:59 +0200
Subject: [Borgbackup] Moving from old to new backup server
In-Reply-To: <999439927.20210918114431@am-soft.de>
References: <c0856ff2-838b-8527-a4f0-a21028723e9e@dom.de>
 <999439927.20210918114431@am-soft.de>
Message-ID: <2895885e-d8ed-cfac-ed1d-15fba02e62ec@dom.de>

Thanks for your reply!

>> Otherwise I could set up an "empty" borg-VM and move all the clients to
>> the new one. After a couple of weeks when the backups are old and
>> obsolete, I could switch off the old borg server.
> 
> BorgMatic might have an interesting different idea as well: Keep the
> setup as-is and start to backup into multiple repos, some hosted on
> your new server. If your old system really can't be used anymore or
> the backups on the new system are simply enough, just switch your VM.
> 
> https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/

I looked into this already. Redundancy would be nice but I think having
twice as much traffic/backup volume takes too long. So for now it is
just about moving from one to another.

> This would prevent RSYNC-times, creating a new VM and your restore
> etc. could be kept as-is as long as storage is available on the old
> server. OTOH, backup times increase from my understanding.

True but then I'd have to change every client plus keeping the old
scripts and passphrases for a while.

>> I like the other idea more though. What do you think?
> 
> The main point might be if RSYNC is able to catch up with changes,
> like when pruning archives. From my understanding, compacting results
> in new additional files, which might make RSYNC transfer files today
> which are ocmpacted and removed tomorrow in favour of additionally to
> transfer files. If that happens more often than RSYNC is able to
> transfer data, your migration might never finish at all in theory. :-)

I'd rather do a first rsync then make sure that no backups are running
and then a second rsync. That might cost a day without backups but
therefore no rsync-confusion ;-)

> Though, I guess that's pretty unlikely. If I was you, I would decide
> based on how wortht he old backups in the new systems are and how long
> backing up to two destinations might take. Because it might be of
> benefit to really start over with new repos with new default settings
> and stuff.

I see the advantages but rsync + VM-migration is the easier way. Still
didn't made up my mind yet.

> Otherwise, your RSYNC-approach reads fine to me and I would most
> likely do the same.
> 
>> Oh, another question. What would be the best file system? Just ext4 or
>> better something fancier like btfs or xfs? zfs would be nonsense as the
>> raid is handled by hardware.
> 
> ZFS is not just about RAID, just as BTRFS is not only about non-RAID.
> In fact, ZFS seems abit more mature, e.g. cares about things like file
> name encoding instead of simply storing bytes and stuff like that. In
> my opinion, it has the better interface and some better concepts like
> pools vs. datasets vs. snapshots vs. clones etc. as well. There might
> even be better tooling like zfs-auto-snaps, ZREP and stuff like that.
> Depending on where you run your hardware, built-in encryption might be
> a good argument as well, it is for me.

Yes, I agree.

> Said that, have a look at your distribution first and which file
> system it prefers and is best integrated with. Think of updates and
> problems during those, some distributions integrate tightly with
> BTRFS, like SUSE, some chose ZFS instead, like Ubuntu, and provide a
> lot of default tools and setup for either of both. Not only because of
> updates, but backup as well.

Unfortunately xcp-ng which is based on centos7.5 doesn't really support
neither btrfs nor zfs as it looks. I expected at least btrfs or zfs.
A storage repository (SR) based on zfs is possible though but that's not
what I have in mind.

> For me, it's always BTRFS or ZFS these days, depending on the used OS,
> with ZFS being in favor. XFS is in the process of catching up only and
> lacks a lot of tooling and features, ext4 lacks essential concepts
> like snapshots, compression etc. and seems pretty legacy to me. No
> reason to start new systems with the last two mentioned in my opinion.

Unless your OS stands in the way...

I'll do some tests and describe how I eventually succeeded. Might be
helpful for others.

Regards,

Oliver


-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210919/35918a3d/attachment.sig>

From hunguponcontent at gmail.com  Sun Sep 26 12:34:35 2021
From: hunguponcontent at gmail.com (Default User)
Date: Sun, 26 Sep 2021 12:34:35 -0400
Subject: [Borgbackup] default list of exclusions
Message-ID: <CAMaNm6Ee3y_VeFZrMw62FSUN0KYTGtHxqiav_HSXHqXM_3--Eg@mail.gmail.com>

Hello!

I want to try using borgbackup to do backups of my (only) user directory:
/home/debian-user

I just want to do so using Vorta, a GUI for borgbackup.

But I just need a good, general list of directory and file type
exclusions that I can just cut and paste into the Exclude Patterns
window in Vorta.  Something like the  that
appears by default in the Backintime backup program, or a similar one
that is used by default in deja-dup.

Note 1: borgbackup uses a matching pattern called "Fnmatch" with which
I am not familiar, and don't want to learn by trial and error, losing
data in the process.  Which is why I am looking for a "drop-in" basic
exclude list.

Note 2: I am not intending to use borgbackup to back up the whole
system; just /home/debian-user and its subdirectories.  I am using
timeshift to back up the rest of the system.

From lazyvirus at gmx.com  Sun Sep 26 12:53:48 2021
From: lazyvirus at gmx.com (Bzzzz)
Date: Sun, 26 Sep 2021 18:53:48 +0200
Subject: [Borgbackup] default list of exclusions
In-Reply-To: <CAMaNm6Ee3y_VeFZrMw62FSUN0KYTGtHxqiav_HSXHqXM_3--Eg@mail.gmail.com>
References: <CAMaNm6Ee3y_VeFZrMw62FSUN0KYTGtHxqiav_HSXHqXM_3--Eg@mail.gmail.com>
Message-ID: <20210926185348.24f86fc6@msi.defcon1.lan>

On Sun, 26 Sep 2021 12:34:35 -0400
Default User <hunguponcontent at gmail.com> wrote:

> Hello!

Hi,

> I want to try using borgbackup to do backups of my (only) user
> directory: /home/debian-user
> 
> I just want to do so using Vorta, a GUI for borgbackup.
> 
> But I just need a good, general list of directory and file type
> exclusions that I can just cut and paste into the Exclude Patterns
> window in Vorta.  Something like the  that
> appears by default in the Backintime backup program, or a similar one
> that is used by default in deja-dup.

I use simple rules, but some needs to be doubled (to avoid matching
hidden files or dirs), backup is made like this :

borg create \
--verbose \
--exclude-caches \
--show-rc \
--warning \
--filter AME \
--list \
--stats \
--checkpoint-interval 600 \
--comment " ${1}" \
--chunker-params 16,23,16,4095 \
--compression auto,zlib,6 \
--exclude-from /usr/local/sbin/BORG_EXCLUSIONS.list \
::$ARCH /

and here's a sample of /usr/local/sbin/BORG_EXCLUSIONS.list :

/lost+found/*
/lost+found/.?*
/BORG/*
/BORG/.?*
/dev/*
/dev/.?*
/home/*/.gvfs/*
/home/*/.gvfs/.?*
/root/.gvfs/*
/root/.gvfs/.?*
/home/*/.cache/chromium/*
/home/*/.cache/mozilla/firefox/*
/home/*/.claws-mail/tmp/*
/home/*/.googleearth/Cache/*
/home/*/Videos/*
/home/*/Videos/.?*
[?]

Even if you just have one user, this may change in the future, so using
the * wildcard is a good way to fire and forget about that.

> Note 2: I am not intending to use borgbackup to back up the whole
> system; just /home/debian-user and its subdirectories.  I am using
> timeshift to back up the rest of the system.

It doesn't take much time, except for the first backup of course and the
advantage of backing up the whole system in one palce is to be able to
recover from a full failure (like rm -r * from the system root or a disk
failure).

I've tested it with the systemrescue live CD that includes borgbackup in
it's programs, works like a charm and quite fast - so, think twice
before dismissing this possibility.

Jean-Yves

From oh at dom.de  Mon Sep 27 09:17:14 2021
From: oh at dom.de (Oliver Hoffmann)
Date: Mon, 27 Sep 2021 15:17:14 +0200
Subject: [Borgbackup] Moving from old to new backup server
In-Reply-To: <2895885e-d8ed-cfac-ed1d-15fba02e62ec@dom.de>
References: <c0856ff2-838b-8527-a4f0-a21028723e9e@dom.de>
 <999439927.20210918114431@am-soft.de>
 <2895885e-d8ed-cfac-ed1d-15fba02e62ec@dom.de>
Message-ID: <0c4a93f9-c55a-06eb-91f1-d93d3bde6544@dom.de>

Hi all,


so, I decided to do it the boring but save way ;-)

Means no zfs or btrfs and no rsync and VM migration. Instead a raid6
with ext4, a new VM and moving the clients step by step from the old
borg-server to the new one. After 4-6 weeks the backups on the old one
will be obsolete and the whole process is complete.

Thank you for your advise!

Oliver

> Thanks for your reply!
> 
>>> Otherwise I could set up an "empty" borg-VM and move all the clients to
>>> the new one. After a couple of weeks when the backups are old and
>>> obsolete, I could switch off the old borg server.
>>
>> BorgMatic might have an interesting different idea as well: Keep the
>> setup as-is and start to backup into multiple repos, some hosted on
>> your new server. If your old system really can't be used anymore or
>> the backups on the new system are simply enough, just switch your VM.
>>
>> https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/
> 
> I looked into this already. Redundancy would be nice but I think having
> twice as much traffic/backup volume takes too long. So for now it is
> just about moving from one to another.
> 
>> This would prevent RSYNC-times, creating a new VM and your restore
>> etc. could be kept as-is as long as storage is available on the old
>> server. OTOH, backup times increase from my understanding.
> 
> True but then I'd have to change every client plus keeping the old
> scripts and passphrases for a while.
> 
>>> I like the other idea more though. What do you think?
>>
>> The main point might be if RSYNC is able to catch up with changes,
>> like when pruning archives. From my understanding, compacting results
>> in new additional files, which might make RSYNC transfer files today
>> which are ocmpacted and removed tomorrow in favour of additionally to
>> transfer files. If that happens more often than RSYNC is able to
>> transfer data, your migration might never finish at all in theory. :-)
> 
> I'd rather do a first rsync then make sure that no backups are running
> and then a second rsync. That might cost a day without backups but
> therefore no rsync-confusion ;-)
> 
>> Though, I guess that's pretty unlikely. If I was you, I would decide
>> based on how wortht he old backups in the new systems are and how long
>> backing up to two destinations might take. Because it might be of
>> benefit to really start over with new repos with new default settings
>> and stuff.
> 
> I see the advantages but rsync + VM-migration is the easier way. Still
> didn't made up my mind yet.
> 
>> Otherwise, your RSYNC-approach reads fine to me and I would most
>> likely do the same.
>>
>>> Oh, another question. What would be the best file system? Just ext4 or
>>> better something fancier like btfs or xfs? zfs would be nonsense as the
>>> raid is handled by hardware.
>>
>> ZFS is not just about RAID, just as BTRFS is not only about non-RAID.
>> In fact, ZFS seems abit more mature, e.g. cares about things like file
>> name encoding instead of simply storing bytes and stuff like that. In
>> my opinion, it has the better interface and some better concepts like
>> pools vs. datasets vs. snapshots vs. clones etc. as well. There might
>> even be better tooling like zfs-auto-snaps, ZREP and stuff like that.
>> Depending on where you run your hardware, built-in encryption might be
>> a good argument as well, it is for me.
> 
> Yes, I agree.
> 
>> Said that, have a look at your distribution first and which file
>> system it prefers and is best integrated with. Think of updates and
>> problems during those, some distributions integrate tightly with
>> BTRFS, like SUSE, some chose ZFS instead, like Ubuntu, and provide a
>> lot of default tools and setup for either of both. Not only because of
>> updates, but backup as well.
> 
> Unfortunately xcp-ng which is based on centos7.5 doesn't really support
> neither btrfs nor zfs as it looks. I expected at least btrfs or zfs.
> A storage repository (SR) based on zfs is possible though but that's not
> what I have in mind.
> 
>> For me, it's always BTRFS or ZFS these days, depending on the used OS,
>> with ZFS being in favor. XFS is in the process of catching up only and
>> lacks a lot of tooling and features, ext4 lacks essential concepts
>> like snapshots, compression etc. and seems pretty legacy to me. No
>> reason to start new systems with the last two mentioned in my opinion.
> 
> Unless your OS stands in the way...
> 
> I'll do some tests and describe how I eventually succeeded. Might be
> helpful for others.
> 
> Regards,
> 
> Oliver
> 
> 



-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 203 bytes
Desc: OpenPGP digital signature
URL: <https://mail.python.org/pipermail/borgbackup/attachments/20210927/369f1ba1/attachment.sig>