From david_martin at fastmail.com Sat Apr 1 04:35:47 2017 From: david_martin at fastmail.com (David Martin) Date: Sat, 01 Apr 2017 19:35:47 +1100 Subject: [Borgbackup] 'borg create' fails with whitespaces in directory names [OS X] In-Reply-To: References: <1490853445.1226750.928243408.67690F3B@webmail.messagingengine.com> Message-ID: <1491035747.2007393.930657312.02A3ED86@webmail.messagingengine.com> Hi, On Fri, 31 Mar 2017, at 02:16 AM, Adrian Klaver wrote: > > Not sure it is the whitespace: > > aklaver at tito:~> borg_new create --dry-run -v --stats borg_test::test_run > 'home/aklaver/test space/' > home/aklaver/test space: [Errno 2] No such file or directory: > 'home/aklaver/test space' > > aklaver at tito:~> borg_new create --dry-run -v --stats borg_test::test_run > '/home/aklaver/test space/' Good point, I'm getting the same error when there are no whitespaces in the directory. I'll do a bit more digging! Basically I was just following the example in the docs [0]. [0] https://borgbackup.readthedocs.io/en/stable/quickstart.html#automating-backups From david_martin at fastmail.com Sat Apr 1 05:25:11 2017 From: david_martin at fastmail.com (David Martin) Date: Sat, 01 Apr 2017 20:25:11 +1100 Subject: [Borgbackup] 'borg create' fails with whitespaces in directory names [OS X] In-Reply-To: <1491035747.2007393.930657312.02A3ED86@webmail.messagingengine.com> References: <1490853445.1226750.928243408.67690F3B@webmail.messagingengine.com> <1491035747.2007393.930657312.02A3ED86@webmail.messagingengine.com> Message-ID: <1491038711.2016824.930680544.28153A6F@webmail.messagingengine.com> On Sat, 1 Apr 2017, at 07:35 PM, David Martin wrote: > On Fri, 31 Mar 2017, at 02:16 AM, Adrian Klaver wrote: > > > > Not sure it is the whitespace: > > > > aklaver at tito:~> borg_new create --dry-run -v --stats borg_test::test_run > > 'home/aklaver/test space/' > > home/aklaver/test space: [Errno 2] No such file or directory: > > 'home/aklaver/test space' > > > > aklaver at tito:~> borg_new create --dry-run -v --stats borg_test::test_run > > '/home/aklaver/test space/' > > Good point, I'm getting the same error when there are no whitespaces in > the directory. I'll do a bit more digging! > Basically I was just following the example in the docs [0]. > > [0] > https://borgbackup.readthedocs.io/en/stable/quickstart.html#automating-backups Well, this is awkward. I was missing the leading / in the path. No wonder it could not locate anything under 'User', as it is '/User'. Seems to be working fine now. Thanks for you help! From maurice.libes at osupytheas.fr Wed Apr 5 10:47:16 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Wed, 5 Apr 2017 16:47:16 +0200 Subject: [Borgbackup] fuse problem Message-ID: hi to all It is not directly a borg problem, but i am bored because i can not acces any more to the archive backup On a particular debian8 machine, the borg mount command fails $ borg mount /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore fuse: device not found, try 'modprobe fuse' first root at rancid:~/BORG# modprobe fuse modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' I have however installed the llfuse python package with pip3 install llfuse root at rancid:~/BORG# pip3 list | grep fuse llfuse (1.2) If you know the issue and a solution ? thank you ML -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at enkore.de Wed Apr 5 11:02:47 2017 From: public at enkore.de (Marian Beermann) Date: Wed, 5 Apr 2017 17:02:47 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: References: Message-ID: <689efeb1-d0e9-93c4-9c05-afefaa0af3cf@enkore.de> Perhaps a mismatched kernel vs. what's installed. I usually get similar errors when trying to load modules on Arch Linux after upgrading the kernel, since upgrading the kernel removes the modules directory of the currently running one. Or maybe the file mentioned there is corrupted (I/O error), or got lost. Try reinstalling all the fuse packages from the operating system. Cheers, Marian On 05.04.2017 16:47, Maurice Libes wrote: > hi to all > > It is not directly a borg problem, but i am bored because i can not > acces any more to the archive backup > On a particular debian8 machine, the borg mount command fails > > $ borg mount /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore > > fuse: device not found, try 'modprobe fuse' first > > root at rancid:~/BORG# modprobe fuse > > modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' > > I have however installed the llfuse python package with pip3 install llfuse > > root at rancid:~/BORG# pip3 list | grep fuse > > llfuse (1.2) > > > > If you know the issue and a solution ? thank you > > ML > > > > -- > M. LIBES > Service Informatique OSU Pytheas - UMS 3470 CNRS > Batiment Oceanomed > Campus de Luminy > 13288 Marseille cedex 9 > Tel: 04860 90529 > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From tw at waldmann-edv.de Wed Apr 5 10:58:54 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 5 Apr 2017 16:58:54 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: References: Message-ID: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> > root at rancid:~/BORG# modprobe fuse > modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' Does running depmod -a and rebooting fix this? This is a kernel module problem. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From adrian.klaver at aklaver.com Wed Apr 5 12:04:57 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Wed, 5 Apr 2017 09:04:57 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: References: Message-ID: <40df57fb-2d9d-3488-9870-dcc8f3beacc5@aklaver.com> On 04/05/2017 07:47 AM, Maurice Libes wrote: > hi to all > > It is not directly a borg problem, but i am bored because i can not > acces any more to the archive backup > On a particular debian8 machine, the borg mount command fails > > $ borg mount /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore > > fuse: device not found, try 'modprobe fuse' first > > root at rancid:~/BORG# modprobe fuse > > modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' > > I have however installed the llfuse python package with pip3 install llfuse > > root at rancid:~/BORG# pip3 list | grep fuse > > llfuse (1.2) > > Have you installed the Debian fuse package? > > If you know the issue and a solution ? thank you > > ML > > > > -- > M. LIBES > Service Informatique OSU Pytheas - UMS 3470 CNRS > Batiment Oceanomed > Campus de Luminy > 13288 Marseille cedex 9 > Tel: 04860 90529 > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Thu Apr 6 08:46:53 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 6 Apr 2017 14:46:53 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> Message-ID: <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> Le 05/04/2017 ? 16:58, Thomas Waldmann a ?crit : >> root at rancid:~/BORG# modprobe fuse >> modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' > Does running depmod -a and rebooting fix this? > > This is a kernel module problem. > thanks to all answers, i have removed and re-installed llfuse - from pip3 install llfuse and also via - apt-get install python3-llfuse but when borg moutinf the archives it is the same "fuse: device not found, try 'modprobe fuse' first So llfuse seems to need some kernel modules in order to work ? but these modules are not installed on a LXC virtual machine on which I have installed the borg server the "borg mount" problem occurs on a proxmox LXC virtual machine, which lacks some kernel modules such as /lib/modules/4.4.6-1-pve/modules.dep.bin" I don't know how to install these kernel modules on the LXC virtual machine (as they are shared with the physical host) maybe borg don't have to work on a LXC VM as concerning mounting the archives ? Anyway I can acces the archives from another client PC, in which I share the same borg user and same borg passphrase borg mount borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 /mnt/provigo-borg/ I don't know if it is a good practice to acces to the archives, from differents client PC, but it works advices ? ML -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From tw at waldmann-edv.de Thu Apr 6 09:08:50 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 6 Apr 2017 15:08:50 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> Message-ID: <21dcd683-20ca-53ba-745b-e38ae8f3ca90@waldmann-edv.de> > I don't know how to install these kernel modules on the LXC virtual > machine (as they are shared with the physical host) I'ld guess you need to install the required kernel modules on the host then. Likely, they are already there and you just need to do a `modprobe fuse` on the host (and, if that works, add it to some file, so this is done automatically at boot time). It's just a guess. I don't know if LXC has some more fundamental issue regarding kernel modules or fuse support. > Anyway I can acces the archives from another client PC, in which I share > the same borg user > and same borg passphrase > > borg mount > borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 > /mnt/provigo-borg/ > > I don't know if it is a good practice to acces to the archives, from > differents client PC, but it works borg mount is read-only, so it does not modify the repo, no problem with that. just make sure only trusted clients access backup data. also, borg mount locks the repo, so don't have something mounted when the repo shall be used otherwise (e.g. at backup time). -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From adrian.klaver at aklaver.com Thu Apr 6 12:17:00 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Thu, 6 Apr 2017 09:17:00 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> Message-ID: <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> On 04/06/2017 05:46 AM, Maurice Libes wrote: > > > Le 05/04/2017 ? 16:58, Thomas Waldmann a ?crit : >>> root at rancid:~/BORG# modprobe fuse >>> modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could >>> not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' >> Does running depmod -a and rebooting fix this? >> >> This is a kernel module problem. >> > thanks to all answers, > > i have removed and re-installed llfuse > - from pip3 install llfuse > and also via > - apt-get install python3-llfuse > > but when borg moutinf the archives it is the same "fuse: device not > found, try 'modprobe fuse' first > > > So llfuse seems to need some kernel modules in order to work ? > but these modules are not installed on a LXC virtual machine on which I > have installed the borg server > > the "borg mount" problem occurs on a proxmox LXC virtual machine, which > lacks some kernel modules such as /lib/modules/4.4.6-1-pve/modules.dep.bin" https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/ To activate fuse in LXC container, it's necessary to create the corresponding dev node. With root: Code: mknod -m 666 /dev/fuse c 10 229 > > I don't know how to install these kernel modules on the LXC virtual > machine (as they are shared with the physical host) > maybe borg don't have to work on a LXC VM as concerning mounting the > archives ? > > > Anyway I can acces the archives from another client PC, in which I share > the same borg user > and same borg passphrase > > borg mount > borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 > /mnt/provigo-borg/ > > I don't know if it is a good practice to acces to the archives, from > differents client PC, but it works > > advices ? > > ML > > > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Thu Apr 6 12:25:58 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 6 Apr 2017 18:25:58 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> Message-ID: <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> Le 06/04/2017 ? 18:17, Adrian Klaver a ?crit : > On 04/06/2017 05:46 AM, Maurice Libes wrote: >> >> >> Le 05/04/2017 ? 16:58, Thomas Waldmann a ?crit : >>>> root at rancid:~/BORG# modprobe fuse >>>> modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could >>>> not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' >>> Does running depmod -a and rebooting fix this? >>> >>> This is a kernel module problem. >>> >> thanks to all answers, >> >> i have removed and re-installed llfuse >> - from pip3 install llfuse >> and also via >> - apt-get install python3-llfuse >> >> but when borg moutinf the archives it is the same "fuse: device not >> found, try 'modprobe fuse' first >> >> >> So llfuse seems to need some kernel modules in order to work ? >> but these modules are not installed on a LXC virtual machine on which I >> have installed the borg server >> >> the "borg mount" problem occurs on a proxmox LXC virtual machine, which >> lacks some kernel modules such as >> /lib/modules/4.4.6-1-pve/modules.dep.bin" > > https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/ > > To activate fuse in LXC container, it's necessary to create the > corresponding dev node. With root: > Code: > > mknod -m 666 /dev/fuse c 10 229 > > well seen :-) thanks now I have the good device... root at rancid:~# ll /dev/fuse crw-rw-rw- 1 root root 10, 229 Apr 6 15:22 /dev/fuse the problem evolves.. it's better but now I have a "permission denied" root at rancid:~# borg mount /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ fuse: mount failed: Permission denied I am searching which permission and where M >> >> I don't know how to install these kernel modules on the LXC virtual >> machine (as they are shared with the physical host) >> maybe borg don't have to work on a LXC VM as concerning mounting the >> archives ? >> >> >> Anyway I can acces the archives from another client PC, in which I share >> the same borg user >> and same borg passphrase >> >> borg mount >> borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 >> >> /mnt/provigo-borg/ >> >> I don't know if it is a good practice to acces to the archives, from >> differents client PC, but it works >> >> advices ? >> >> ML >> >> >> >> >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > > -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From adrian.klaver at aklaver.com Thu Apr 6 12:30:53 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Thu, 6 Apr 2017 09:30:53 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> Message-ID: On 04/06/2017 09:25 AM, Maurice Libes wrote: > > > Le 06/04/2017 ? 18:17, Adrian Klaver a ?crit : >> On 04/06/2017 05:46 AM, Maurice Libes wrote: >>> >>> >>> Le 05/04/2017 ? 16:58, Thomas Waldmann a ?crit : >>>>> root at rancid:~/BORG# modprobe fuse >>>>> modprobe: ERROR: ../libkmod/libkmod.c:557 kmod_search_moddep() could >>>>> not open moddep file '/lib/modules/4.4.6-1-pve/modules.dep.bin' >>>> Does running depmod -a and rebooting fix this? >>>> >>>> This is a kernel module problem. >>>> >>> thanks to all answers, >>> >>> i have removed and re-installed llfuse >>> - from pip3 install llfuse >>> and also via >>> - apt-get install python3-llfuse >>> >>> but when borg moutinf the archives it is the same "fuse: device not >>> found, try 'modprobe fuse' first >>> >>> >>> So llfuse seems to need some kernel modules in order to work ? >>> but these modules are not installed on a LXC virtual machine on which I >>> have installed the borg server >>> >>> the "borg mount" problem occurs on a proxmox LXC virtual machine, which >>> lacks some kernel modules such as >>> /lib/modules/4.4.6-1-pve/modules.dep.bin" >> >> https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/ >> >> To activate fuse in LXC container, it's necessary to create the >> corresponding dev node. With root: >> Code: >> >> mknod -m 666 /dev/fuse c 10 229 >> >> > well seen :-) thanks > > now I have the good device... > > root at rancid:~# ll /dev/fuse > > crw-rw-rw- 1 root root 10, 229 Apr 6 15:22 /dev/fuse > > the problem evolves.. it's better but now I have a "permission denied" > > root at rancid:~# borg mount > /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ > > fuse: mount failed: Permission denied > > I am searching which permission and where My guess is permissions on: /mnt/borg-restore/ > > M >>> >>> I don't know how to install these kernel modules on the LXC virtual >>> machine (as they are shared with the physical host) >>> maybe borg don't have to work on a LXC VM as concerning mounting the >>> archives ? >>> >>> >>> Anyway I can acces the archives from another client PC, in which I share >>> the same borg user >>> and same borg passphrase >>> >>> borg mount >>> borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 >>> >>> /mnt/provigo-borg/ >>> >>> I don't know if it is a good practice to acces to the archives, from >>> differents client PC, but it works >>> >>> advices ? >>> >>> ML >>> >>> >>> >>> >>> >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >>> >> >> > -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Thu Apr 6 12:35:44 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 6 Apr 2017 18:35:44 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> Message-ID: <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> hanks to all answers, >>>> >>>> i have removed and re-installed llfuse >>>> - from pip3 install llfuse >>>> and also via >>>> - apt-get install python3-llfuse >>>> >>>> but when borg moutinf the archives it is the same "fuse: device not >>>> found, try 'modprobe fuse' first >>>> >>>> >>>> So llfuse seems to need some kernel modules in order to work ? >>>> but these modules are not installed on a LXC virtual machine on >>>> which I >>>> have installed the borg server >>>> >>>> the "borg mount" problem occurs on a proxmox LXC virtual machine, >>>> which >>>> lacks some kernel modules such as >>>> /lib/modules/4.4.6-1-pve/modules.dep.bin" >>> >>> https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/ >>> >>> To activate fuse in LXC container, it's necessary to create the >>> corresponding dev node. With root: >>> Code: >>> >>> mknod -m 666 /dev/fuse c 10 229 >>> >>> >> well seen :-) thanks >> >> now I have the good device... >> >> root at rancid:~# ll /dev/fuse >> >> crw-rw-rw- 1 root root 10, 229 Apr 6 15:22 /dev/fuse >> >> the problem evolves.. it's better but now I have a "permission denied" >> >> root at rancid:~# borg mount >> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >> >> fuse: mount failed: Permission denied >> >> I am searching which permission and where > > My guess is permissions on: > > /mnt/borg-restore/ > I would have liked root at rancid:~# ll /mnt/borg-restore/ -d drwxrwxrwx 2 borg root 4096 Apr 3 09:50 /mnt/borg-restore/ root at rancid:~# borg mount /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ fuse: mount failed: Permission denied I touch a particular case, sorry M >> M >>>> >>>> I don't know how to install these kernel modules on the LXC virtual >>>> machine (as they are shared with the physical host) >>>> maybe borg don't have to work on a LXC VM as concerning mounting the >>>> archives ? >>>> >>>> >>>> Anyway I can acces the archives from another client PC, in which I >>>> share >>>> the same borg user >>>> and same borg passphrase >>>> >>>> borg mount >>>> borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 >>>> >>>> >>>> /mnt/provigo-borg/ >>>> >>>> I don't know if it is a good practice to acces to the archives, from >>>> differents client PC, but it works >>>> >>>> advices ? >>>> >>>> ML >>>> >>>> >>>> >>>> >>>> >>>> _______________________________________________ >>>> Borgbackup mailing list >>>> Borgbackup at python.org >>>> https://mail.python.org/mailman/listinfo/borgbackup >>>> >>> >>> >> > > -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From adrian.klaver at aklaver.com Thu Apr 6 12:42:13 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Thu, 6 Apr 2017 09:42:13 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> Message-ID: <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> On 04/06/2017 09:35 AM, Maurice Libes wrote: > hanks to all answers, >>>>> >>>>> i have removed and re-installed llfuse >>>>> - from pip3 install llfuse >>>>> and also via >>>>> - apt-get install python3-llfuse >>>>> >>>>> but when borg moutinf the archives it is the same "fuse: device not >>>>> found, try 'modprobe fuse' first >>>>> >>>>> >>>>> So llfuse seems to need some kernel modules in order to work ? >>>>> but these modules are not installed on a LXC virtual machine on >>>>> which I >>>>> have installed the borg server >>>>> >>>>> the "borg mount" problem occurs on a proxmox LXC virtual machine, >>>>> which >>>>> lacks some kernel modules such as >>>>> /lib/modules/4.4.6-1-pve/modules.dep.bin" >>>> >>>> https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/ >>>> >>>> To activate fuse in LXC container, it's necessary to create the >>>> corresponding dev node. With root: >>>> Code: >>>> >>>> mknod -m 666 /dev/fuse c 10 229 >>>> >>>> >>> well seen :-) thanks >>> >>> now I have the good device... >>> >>> root at rancid:~# ll /dev/fuse >>> >>> crw-rw-rw- 1 root root 10, 229 Apr 6 15:22 /dev/fuse >>> >>> the problem evolves.. it's better but now I have a "permission denied" >>> >>> root at rancid:~# borg mount >>> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >>> >>> fuse: mount failed: Permission denied >>> >>> I am searching which permission and where >> >> My guess is permissions on: >> >> /mnt/borg-restore/ >> > I would have liked > > root at rancid:~# ll /mnt/borg-restore/ -d > > drwxrwxrwx 2 borg root 4096 Apr 3 09:50 /mnt/borg-restore/ > > root at rancid:~# borg mount > /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ > > fuse: mount failed: Permission denied What are permissions on?: /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 I generally try to stay away from running as root for user applications. In my case my repos are in directories accessible by my user name (aklaver) and the mount point I use is in my user directory. I run borg as the aklaver user. > > I touch a particular case, sorry I don't know what you mean above? > M > >>> M >>>>> >>>>> I don't know how to install these kernel modules on the LXC virtual >>>>> machine (as they are shared with the physical host) >>>>> maybe borg don't have to work on a LXC VM as concerning mounting the >>>>> archives ? >>>>> >>>>> >>>>> Anyway I can acces the archives from another client PC, in which I >>>>> share >>>>> the same borg user >>>>> and same borg passphrase >>>>> >>>>> borg mount >>>>> borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 >>>>> >>>>> >>>>> /mnt/provigo-borg/ >>>>> >>>>> I don't know if it is a good practice to acces to the archives, from >>>>> differents client PC, but it works >>>>> >>>>> advices ? >>>>> >>>>> ML >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> _______________________________________________ >>>>> Borgbackup mailing list >>>>> Borgbackup at python.org >>>>> https://mail.python.org/mailman/listinfo/borgbackup >>>>> >>>> >>>> >>> >> >> > -- Adrian Klaver adrian.klaver at aklaver.com From adrian.klaver at aklaver.com Thu Apr 6 15:37:43 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Thu, 6 Apr 2017 12:37:43 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> Message-ID: <1a1b05bb-57be-6e02-8273-eecc7c495a6f@aklaver.com> On 04/06/2017 09:35 AM, Maurice Libes wrote: > hanks to all answers, >>>>> >>>>> i have removed and re-installed llfuse >>>>> - from pip3 install llfuse >>>>> and also via >>>>> - apt-get install python3-llfuse >>>>> >>>>> but when borg moutinf the archives it is the same "fuse: device not >>>>> found, try 'modprobe fuse' first >>>>> >>>>> >>>>> So llfuse seems to need some kernel modules in order to work ? >>>>> but these modules are not installed on a LXC virtual machine on >>>>> which I >>>>> have installed the borg server >>>>> >>>>> the "borg mount" problem occurs on a proxmox LXC virtual machine, >>>>> which >>>>> lacks some kernel modules such as >>>>> /lib/modules/4.4.6-1-pve/modules.dep.bin" >>>> >>>> https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/ >>>> >>>> To activate fuse in LXC container, it's necessary to create the >>>> corresponding dev node. With root: >>>> Code: >>>> >>>> mknod -m 666 /dev/fuse c 10 229 >>>> >>>> >>> well seen :-) thanks >>> >>> now I have the good device... >>> >>> root at rancid:~# ll /dev/fuse >>> >>> crw-rw-rw- 1 root root 10, 229 Apr 6 15:22 /dev/fuse >>> >>> the problem evolves.. it's better but now I have a "permission denied" >>> >>> root at rancid:~# borg mount >>> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >>> >>> fuse: mount failed: Permission denied >>> >>> I am searching which permission and where >> >> My guess is permissions on: >> >> /mnt/borg-restore/ >> > I would have liked > > root at rancid:~# ll /mnt/borg-restore/ -d > > drwxrwxrwx 2 borg root 4096 Apr 3 09:50 /mnt/borg-restore/ > > root at rancid:~# borg mount > /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ > > fuse: mount failed: Permission denied Thought of something else. Could be a security framework e.g. SELinux/AppArmor preventing access to the mount point. You might want to take a look in the system logs to see if they might useful information. > > I touch a particular case, sorry > M > >>> M -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Fri Apr 7 04:31:25 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Fri, 7 Apr 2017 10:31:25 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> Message-ID: >>>> the problem evolves.. it's better but now I have a "permission denied" >>>> >>>> root at rancid:~# borg mount >>>> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >>>> >>>> fuse: mount failed: Permission denied >>>> >>>> I am searching which permission and where >>> >>> My guess is permissions on: >>> >>> /mnt/borg-restore/ >>> >> I would have liked >> >> root at rancid:~# ll /mnt/borg-restore/ -d >> >> drwxrwxrwx 2 borg root 4096 Apr 3 09:50 /mnt/borg-restore/ >> >> root at rancid:~# borg mount >> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >> >> fuse: mount failed: Permission denied > > What are permissions on?: > > /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 > root at rancid:~/BORG# ll /mnt/provigo-borg/sauve-bioinfo total 170658 -rw------- 1 borg borg 26 Feb 28 16:56 README -rw------- 1 borg borg 635 Feb 28 16:56 config drwx------ 70 borg borg 70 Mar 30 02:27 data -rw------- 1 borg borg 2760423 Apr 7 02:23 hints.678061 -rw------- 1 borg borg 171652778 Apr 7 02:23 index.678061 > I generally try to stay away from running as root for user > applications. In my case my repos are in directories accessible by my > user name (aklaver) and the mount point I use is in my user directory. > I run borg as the aklaver user. > yes it's a little bit confused for me.. my general case is to make PC backup via the LAN with a borg user... like that / $ borg create -v --stats borg at myserver.fr:/mnt/provigo-borg/sauve-pc::pcsargasse-2017-03-13// / so the archive belongs to borg.. and I can mount the archive on the PC that case works fine// but I test another case described above which causes problem at mounting time... I want to make a backup directly on the borg server itself (which is a LXC vm) between 2 NAS ... I backup some big directories from one NAS to another NAS via borg so in that case, the backup is made with the "root" user... like that /$ borg create --stats --compression zlib,3 /mnt/provigo-borg/sauve-netappdata::netappdata-2017-04-06 / in that case the archive belong to root root at rancid:~# ll /mnt/provigo-borg/sauve-netappdata/total 10478 -rw------- 1 root borg 26 Apr 3 17:37 README -rw------- 1 root borg 655 Apr 3 17:37 config drwx------ 5 root borg 5 Apr 6 18:36 data -rw------- 1 root borg 78759 Apr 6 20:22 hints.26349 -rw------- 1 root borg 10485898 Apr 6 20:22 index.26349 and the problem is just to mount the archive on this LXC borg server (fuse problem) $ /usr/local/bin/borg mount /mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-06 /mnt/borg-restore/ fuse: mount failed: Permission denied I didn't find the solution yet, why the mounting is denied thanks ML >> >> I touch a particular case, sorry > > I don't know what you mean above? > >> M >> >>>> M >>>>>> >>>>>> I don't know how to install these kernel modules on the LXC virtual >>>>>> machine (as they are shared with the physical host) >>>>>> maybe borg don't have to work on a LXC VM as concerning mounting >>>>>> the >>>>>> archives ? >>>>>> >>>>>> >>>>>> Anyway I can acces the archives from another client PC, in which I >>>>>> share >>>>>> the same borg user >>>>>> and same borg passphrase >>>>>> >>>>>> borg mount >>>>>> borg at myserver.fr:/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-05 >>>>>> >>>>>> >>>>>> >>>>>> /mnt/provigo-borg/ >>>>>> >>>>>> I don't know if it is a good practice to acces to the archives, from >>>>>> differents client PC, but it works >>>>>> >>>>>> advices ? >>>>>> >>>>>> ML >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> Borgbackup mailing list >>>>>> Borgbackup at python.org >>>>>> https://mail.python.org/mailman/listinfo/borgbackup >>>>>> >>>>> >>>>> >>>> >>> >>> >> > > -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From adrian.klaver at aklaver.com Fri Apr 7 10:32:09 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Fri, 7 Apr 2017 07:32:09 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> Message-ID: On 04/07/2017 01:31 AM, Maurice Libes wrote: > >>>>> the problem evolves.. it's better but now I have a "permission denied" >>>>> >>>>> root at rancid:~# borg mount >>>>> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >>>>> >>>>> fuse: mount failed: Permission denied >>>>> >>>>> I am searching which permission and where >>>> >>>> My guess is permissions on: >>>> >>>> /mnt/borg-restore/ >>>> >>> I would have liked >>> >>> root at rancid:~# ll /mnt/borg-restore/ -d >>> >>> drwxrwxrwx 2 borg root 4096 Apr 3 09:50 /mnt/borg-restore/ >>> >>> root at rancid:~# borg mount >>> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 /mnt/borg-restore/ >>> >>> fuse: mount failed: Permission denied >> >> What are permissions on?: >> >> /mnt/provigo-borg/sauve-bioinfo::bioinfo-2017-04-03 >> > root at rancid:~/BORG# ll /mnt/provigo-borg/sauve-bioinfo > > total 170658 > > -rw------- 1 borg borg 26 Feb 28 16:56 README > > -rw------- 1 borg borg 635 Feb 28 16:56 config > > drwx------ 70 borg borg 70 Mar 30 02:27 data > > -rw------- 1 borg borg 2760423 Apr 7 02:23 hints.678061 > > -rw------- 1 borg borg 171652778 Apr 7 02:23 index.678061 > > >> I generally try to stay away from running as root for user >> applications. In my case my repos are in directories accessible by my >> user name (aklaver) and the mount point I use is in my user directory. >> I run borg as the aklaver user. >> > yes it's a little bit confused for me.. > my general case is to make PC backup via the LAN with a borg user... > like that > / > $ borg create -v --stats > borg at myserver.fr:/mnt/provigo-borg/sauve-pc::pcsargasse-2017-03-13// > / > so the archive belongs to borg.. and I can mount the archive on the PC > that case works fine// > > but I test another case described above which causes problem at mounting > time... > > I want to make a backup directly on the borg server itself (which is a > LXC vm) between 2 NAS ... > I backup some big directories from one NAS to another NAS via borg Somewhat confused, are you backing up to the borg server(LXC vm) or to the NAS? Or are you saying the NAS you are backing up to is mounted on the LXC vm? > > so in that case, the backup is made with the "root" user... like that > /$ borg create --stats --compression zlib,3 > /mnt/provigo-borg/sauve-netappdata::netappdata-2017-04-06 > / > in that case the archive belong to root > > root at rancid:~# ll /mnt/provigo-borg/sauve-netappdata/total 10478 > > -rw------- 1 root borg 26 Apr 3 17:37 README > > -rw------- 1 root borg 655 Apr 3 17:37 config > > drwx------ 5 root borg 5 Apr 6 18:36 data > > -rw------- 1 root borg 78759 Apr 6 20:22 hints.26349 > > -rw------- 1 root borg 10485898 Apr 6 20:22 index.26349 > > > and the problem is just to mount the archive on this LXC borg server > (fuse problem) > > $ /usr/local/bin/borg mount /mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-06 /mnt/borg-restore/ > > fuse: mount failed: Permission denied Seems we have moved on from a FUSE problem to a permissions problem. Can you run other borg commands against the repo, e.g. list or info? > > > I didn't find the solution yet, why the mounting is denied > > thanks > > ML > > >>> >>> I touch a particular case, sorry >> >> I don't know what you mean above? >> >>> M >>> >>>>> M >>>>>>> -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Mon Apr 10 05:30:45 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Mon, 10 Apr 2017 11:30:45 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> Message-ID: >> >>> I generally try to stay away from running as root for user >>> applications. In my case my repos are in directories accessible by my >>> user name (aklaver) and the mount point I use is in my user directory. >>> I run borg as the aklaver user. >>> >> yes it's a little bit confused for me.. >> my general case is to make PC backup via the LAN with a borg user... >> like that >> / >> $ borg create -v --stats >> borg at myserver.fr:/mnt/provigo-borg/sauve-pc::pcsargasse-2017-03-13// >> / >> so the archive belongs to borg.. and I can mount the archive on the PC >> that case works fine// >> >> but I test another case described above which causes problem at mounting >> time... >> >> I want to make a backup directly on the borg server itself (which is a >> LXC vm) between 2 NAS ... >> I backup some big directories from one NAS to another NAS via borg > > Somewhat confused, are you backing up to the borg server(LXC vm) or to > the NAS? to the NAS (wich is mounted on the borg server in "rw") > > Or are you saying the NAS you are backing up to is mounted on the LXC vm? that's it, the NAS is mounted on the LXC vm > >> >> so in that case, the backup is made with the "root" user... like that >> /$ borg create --stats --compression zlib,3 >> /mnt/provigo-borg/sauve-netappdata::netappdata-2017-04-06 >> / >> in that case the archive belong to root >> >> root at rancid:~# ll /mnt/provigo-borg/sauve-netappdata/total 10478 >> -rw------- 1 root borg 26 Apr 3 17:37 README >> -rw------- 1 root borg 655 Apr 3 17:37 config >> drwx------ 5 root borg 5 Apr 6 18:36 data >> -rw------- 1 root borg 78759 Apr 6 20:22 hints.26349 >> -rw------- 1 root borg 10485898 Apr 6 20:22 index.26349 >> >> >> and the problem is just to mount the archive on this LXC borg server >> (fuse problem) >> >> $ /usr/local/bin/borg mount >> /mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-06 >> /mnt/borg-restore/ >> >> fuse: mount failed: Permission denied > > Seems we have moved on from a FUSE problem to a permissions problem. yes... It has changed since I have created the /dev/fuse device by hand (which needs to be recreated every start of the VM) ll /dev/fuse crw-rw-rw- 1 root root 10, 229 Apr 7 15:12 /dev/fuse to my opinion it rather seems to be a "fuse" permission problem in the LXC context even in the LXC forum, that's not very clear [https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/page-2] I don't succeed to know if mounting a fuse device in a LXC VM is possible or not maybe due to the LXC capabilities? I don't know > > Can you run other borg commands against the repo, e.g. list or info? > absolutely yes root at rancid:~/BORG# borg list myserver.fr:/mnt/provigo-borg/sauve-netappbioinfo ... netapp-bioinfo-2017-04-07 Fri, 2017-04-07 03:00:03 netapp-bioinfo-2017-04-08 Sat, 2017-04-08 03:00:04 netapp-bioinfo-2017-04-09 Sun, 2017-04-09 03:00:09 netapp-bioinfo-2017-04-10 Mon, 2017-04-10 03:00:05 $ borg info myserver.fr:/mnt/provigo-borg/sauve-netappbioinfo::netapp-bioinfo-2017-04-10 Name: netapp-bioinfo-2017-04-10 Fingerprint: 94f2a15b3d117cd5565871e08c77366293b6aa842e3f0454712f5f375a5c624b Hostname: rancid Username: root Time (start): Mon, 2017-04-10 03:00:05 Time (end): Mon, 2017-04-10 03:02:05 Command line: /usr/local/bin/borg create -v --stats --compression zlib,3 /mnt/provigo-borg/sauve-netappbioinfo::netapp-bioinfo-2017-04-10 /mnt/netapp2-DATA/BIOINFO/ --exclude .snapshot/ --exclude *.iso --exclude *.ova --exclude /home/*/.cache Number of files: 67434 Original size Compressed size Deduplicated size This archive: 637.95 GB 178.26 GB 8.75 MB All archives: 8.29 TB 2.32 TB 142.59 GB Unique chunks Total chunks Chunk index: 194965 3724397 >> >> >> I didn't find the solution yet, why the mounting is denied >> >> thanks >> >> ML >> >> >>>> >>>> I touch a particular case, sorry >>> >>> I don't know what you mean above? >>> >>>> M >>>> >>>>>> M >>>>>>>> > > > -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From tw at waldmann-edv.de Mon Apr 10 08:07:41 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 10 Apr 2017 14:07:41 +0200 Subject: [Borgbackup] mailinglist etiquette / style reminder Message-ID: <6a3c1158-4be2-cf60-a9a1-39cf84625295@waldmann-edv.de> Dear mailing list users, I noticed that the style of the postings to this list recently left a lot to be desired. Just as a short reminder (please google yourself for longer mailing list style / etiquette writeups if you did not read one yet): It is usually not neccessary nor good style to completely quote a previous message - one should only quote the part one is responding to (so that it is clear to what the response is referring to). One shall remove everything that is not needed for that purpose. It usually should look like this: ------------------------------------------------------ > part of the previous post you are addressing your response BELOW the stuff you are addressing. > another part of the previous post you are addressing your response to that other part. ------------------------------------------------------ The reason for this is that you do not want to waste the readers' time they would need to locate the essential (new) part of your message between a lot of old stuff they've already read. Thanks for writing nice posts. Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From adrian.klaver at aklaver.com Mon Apr 10 10:14:22 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Mon, 10 Apr 2017 07:14:22 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> Message-ID: <10298e99-27ee-38b1-88cb-c862319570cc@aklaver.com> On 04/10/2017 02:30 AM, Maurice Libes wrote: > >>> >>>> I generally try to stay away from running as root for user >> >> Somewhat confused, are you backing up to the borg server(LXC vm) or to >> the NAS? > to the NAS (wich is mounted on the borg server in "rw") >> >> Or are you saying the NAS you are backing up to is mounted on the LXC vm? > that's it, the NAS is mounted on the LXC vm > >> >>> >> Seems we have moved on from a FUSE problem to a permissions problem. > yes... > > It has changed since I have created the /dev/fuse device by hand (which > needs to be recreated every start of the VM) That would seem to call for a script. > > ll /dev/fuse > > crw-rw-rw- 1 root root 10, 229 Apr 7 15:12 /dev/fuse > > > to my opinion it rather seems to be a "fuse" permission problem in the > LXC context > > even in the LXC forum, that's not very clear > [https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/page-2] > I don't succeed to know if mounting a fuse device in a LXC VM is > possible or not > > maybe due to the LXC capabilities? I don't know Could be I don't know as I do not use LXC. I would suggest asking for advice on the forum linked to above. > >> >> Can you run other borg commands against the repo, e.g. list or info? >> > absolutely yes So Borg works with the exception of mount. -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Mon Apr 10 10:31:56 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Mon, 10 Apr 2017 16:31:56 +0200 Subject: [Borgbackup] fuse problem In-Reply-To: <10298e99-27ee-38b1-88cb-c862319570cc@aklaver.com> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> <10298e99-27ee-38b1-88cb-c862319570cc@aklaver.com> Message-ID: <16667fb5-e9fb-58ba-a328-14a71e3b0a68@osupytheas.fr> >>> >>> Somewhat confused, are you backing up to the borg server(LXC vm) or to >>> the NAS? >> to the NAS (wich is mounted on the borg server in "rw") >>> >>> Or are you saying the NAS you are backing up to is mounted on the >>> LXC vm? >> that's it, the NAS is mounted on the LXC vm >> >>> >>>> > >>> Seems we have moved on from a FUSE problem to a permissions problem. >> yes... >> >> It has changed since I have created the /dev/fuse device by hand (which >> needs to be recreated every start of the VM) > > That would seem to call for a script. > >> >> ll /dev/fuse >> >> crw-rw-rw- 1 root root 10, 229 Apr 7 15:12 /dev/fuse >> >> >> to my opinion it rather seems to be a "fuse" permission problem in the >> LXC context >> >> even in the LXC forum, that's not very clear >> [https://forum.proxmox.com/threads/kernel-module-fuse-for-lxc.24855/page-2] >> >> I don't succeed to know if mounting a fuse device in a LXC VM is >> possible or not >> >> maybe due to the LXC capabilities? I don't know > > Could be I don't know as I do not use LXC. I would suggest asking for > advice on the forum linked to above. > >> >>> >>> Can you run other borg commands against the repo, e.g. list or info? >>> >> absolutely yes > > So Borg works with the exception of mount. > > yes! that's not a borg problem, but a fuse problem on a LXC vm, which is annoying since one can not acces to the archive from this LXC VM the workaround is to acces the archive from another computer but that could be a further point of interest to be able to use the whole functionalities of borg on LXC container so a priori nobody on the list use LXC container... you should try ;-) ML -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From adrian.klaver at aklaver.com Mon Apr 10 10:34:41 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Mon, 10 Apr 2017 07:34:41 -0700 Subject: [Borgbackup] fuse problem In-Reply-To: <16667fb5-e9fb-58ba-a328-14a71e3b0a68@osupytheas.fr> References: <6aebbae4-72d7-600e-e74e-1e6e9c5ee4dc@waldmann-edv.de> <57981ea0-0743-57c6-db2e-97cb58a9c5f1@osupytheas.fr> <43930f9f-39af-f9fe-b1b4-ba491ad55cc2@aklaver.com> <089e017d-dc48-07cf-986a-f98a4c1ea404@osupytheas.fr> <560dd41c-cd59-3d11-638c-a09b7cd95a0d@osupytheas.fr> <299cee0f-7930-eceb-a551-1eed8a3fc706@aklaver.com> <10298e99-27ee-38b1-88cb-c862319570cc@aklaver.com> <16667fb5-e9fb-58ba-a328-14a71e3b0a68@osupytheas.fr> Message-ID: <002095ab-b2b2-10ef-7ea6-44cbe7c878f7@aklaver.com> On 04/10/2017 07:31 AM, Maurice Libes wrote: > > >>>> >> >> So Borg works with the exception of mount. >> >> > yes! > > that's not a borg problem, but a fuse problem on a LXC vm, which is > annoying since one can not acces to the archive from this LXC VM > > the workaround is to acces the archive from another computer > > > but that could be a further point of interest to be able to use the > whole functionalities of borg on LXC container > > so a priori nobody on the list use LXC container... you should try ;-) Well I do not use any sort of container, it is a complication that I have not found a use for. > > ML > > -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Thu Apr 13 05:10:24 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 13 Apr 2017 11:10:24 +0200 Subject: [Borgbackup] backup without end Message-ID: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> hello I have launched a backup of a large amount of data about 7 To on wednesday 7th (from NAS /mnt/netapp2-DATA to NAS /mnt/provigo-borg mounted via NFS) /usr/local/bin/borg create -v --stats --compression zlib,3 /mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-07 /mnt/netapp2-DATA/ --exclude .snapshot/ --exclude sambaressources/ --exclude ADACNI/ --exclude CASSIM/ --exclude PARTAGE_MEB/ --exclude BIOINFO/ --exclude .cache my problem is that it never seems to end, still in progress today If I check the backuped volume in the archive, today ... it is 3. To backuped and normally deduplicated $ du -sh /mnt/provigo-borg/sauve-netappdata 3.1T /mnt/provigo-borg/sauve-netappdata I hesitate to stop the backup how can check what is done, and where we are in the backup ? In your sites have you also backuped big amounts of data ? thanks ML -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Thu Apr 13 06:19:35 2017 From: public at enkore.de (Marian Beermann) Date: Thu, 13 Apr 2017 12:19:35 +0200 Subject: [Borgbackup] backup without end In-Reply-To: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> References: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> Message-ID: <18be4122-86f1-e606-b1d0-f5cf4613890b@enkore.de> Send SIGUSR1 or SIGINFO to the borg process (kill(1)). Cheers, Marian From public at enkore.de Thu Apr 13 06:20:54 2017 From: public at enkore.de (Marian Beermann) Date: Thu, 13 Apr 2017 12:20:54 +0200 Subject: [Borgbackup] backup without end In-Reply-To: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> References: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> Message-ID: <52210412-b8e4-a816-f353-8e3b5dfe50ac@enkore.de> Though 3 TB / 6 days is like 5 MB/s which is really darn slow, even for Borg 1.0.x ... slow network or slow CPU maybe? Cheers, Marian From maurice.libes at osupytheas.fr Fri Apr 14 04:57:23 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Fri, 14 Apr 2017 10:57:23 +0200 Subject: [Borgbackup] backup without end In-Reply-To: <18be4122-86f1-e606-b1d0-f5cf4613890b@enkore.de> References: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> <18be4122-86f1-e606-b1d0-f5cf4613890b@enkore.de> Message-ID: <3178bb8b-8faa-efe3-2d41-e7a76a402c0d@osupytheas.fr> so I stopped this backup that did not end, and I understood the problem when I mounted the .checkpoint archive the problem is that the backup has taken the ._/snapshot/_ directories of my NAS However I had taken care to exclude these directories from the backup here is the "create" line of my script... $BORG create \ -v --stats \ $COMPRESSION \ $BORG_ARCHIVE \ $REP_ASAUVEGARDER \ /* --exclude '.snapshot/' \*/ --exclude 'sambaressources/' \ --exclude 'ADACNI/' \ --exclude 'CASSIM/' \ --exclude 'PARTAGE_MEB/' \ --exclude 'BIOINFO/' \ --exclude '.cache' \ >> ${LOG_PATH} 2>&1 and here is the log file Platform: Linux rancid 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 Linux: debian 8.7 Borg: 1.0.9 Python: CPython 3.4.2 PID: 1472 CWD: /root/BORG sys.argv: ['/usr/local/bin/borg', 'create', '-v', '--stats', '--compression', 'zlib,3', '/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-07', '/mnt/netapp2-DATA/',/*'--exclude', '.snapshot/',*/ '--exclude', 'sambaressources/', '--exclude', 'ADACNI/', '--exclude', 'CASSIM/', '--exclude', 'PARTAGE_MEB/', '--exclude', 'BIOINFO/', '--exclude', '.cache'] Do you see an error or mistake? I don't understand why the backup has taken into account the .snapshot directories? how to exclude it correctly? thanks ML > Send SIGUSR1 or SIGINFO to the borg process (kill(1)). > > Cheers, Marian > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Fri Apr 14 05:12:44 2017 From: public at enkore.de (Marian Beermann) Date: Fri, 14 Apr 2017 11:12:44 +0200 Subject: [Borgbackup] backup without end In-Reply-To: <3178bb8b-8faa-efe3-2d41-e7a76a402c0d@osupytheas.fr> References: <6e4a4351-99dc-ba4c-5073-9906d61ec337@osupytheas.fr> <18be4122-86f1-e606-b1d0-f5cf4613890b@enkore.de> <3178bb8b-8faa-efe3-2d41-e7a76a402c0d@osupytheas.fr> Message-ID: <769312e1-f3f0-b789-dcbc-bd53233201c0@enkore.de> From http://borgbackup.readthedocs.io/en/stable/usage.html#borg-help-patterns > Fnmatch, selector fm: > > This is the default style. > [...] > For a path to match a pattern, it must completely match from start > to end, or must match from the start to just before a path > separator. Except for the root path, paths will never end in the > path separator when matching is attempted. Thus, if a given > pattern ends in a path separator, a ?*? is appended before > matching is attempted. So it'd have to be something like --exclude /mnt/netapp2-DATA/.snapshot/ or --exclude */.snapshot/ (to exclude every directory everywhere named ".snapshot"). Cheers, Marian On 14.04.2017 10:57, Maurice Libes wrote: > so I stopped this backup that did not end, > and I understood the problem when I mounted the .checkpoint archive > the problem is that the backup has taken the ._/snapshot/_ directories > of my NAS > > However I had taken care to exclude these directories from the backup > > here is the "create" line of my script... > > $BORG create \ > -v --stats \ > $COMPRESSION \ > $BORG_ARCHIVE \ > $REP_ASAUVEGARDER \ > /* --exclude '.snapshot/' \*/ > --exclude 'sambaressources/' \ > --exclude 'ADACNI/' \ > --exclude 'CASSIM/' \ > --exclude 'PARTAGE_MEB/' \ > --exclude 'BIOINFO/' \ > --exclude '.cache' \ > >> ${LOG_PATH} 2>&1 > > and here is the log file > > Platform: Linux rancid 4.4.6-1-pve #1 SMP Thu Apr 21 11:25:40 CEST 2016 x86_64 > > Linux: debian 8.7 > > Borg: 1.0.9 Python: CPython 3.4.2 > > PID: 1472 CWD: /root/BORG > > sys.argv: ['/usr/local/bin/borg', 'create', '-v', '--stats', '--compression', 'zlib,3', > '/mnt/provigo-borg/sauve-netappdata::netapp-data-2017-04-07', '/mnt/netapp2-DATA/',/*'--exclude', '.snapshot/',*/ '--exclude', 'sambaressources/', '--exclude', 'ADACNI/', > '--exclude', 'CASSIM/', '--exclude', 'PARTAGE_MEB/', '--exclude', 'BIOINFO/', '--exclude', '.cache'] > > Do you see an error or mistake? > > I don't understand why the backup has taken into account the .snapshot > directories? > how to exclude it correctly? > > thanks > ML > >> Send SIGUSR1 or SIGINFO to the borg process (kill(1)). >> >> Cheers, Marian >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > -- > M. LIBES > Service Informatique OSU Pytheas - UMS 3470 CNRS > Batiment Oceanomed > Campus de Luminy > 13288 Marseille cedex 9 > Tel: 04860 90529 > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From catscrash at catscrash.de Fri Apr 14 07:09:57 2017 From: catscrash at catscrash.de (Catscrash) Date: Fri, 14 Apr 2017 13:09:57 +0200 Subject: [Borgbackup] Speed issues compared with rsync Message-ID: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> Hi, I am trying to switch from a rsync hardlink scenario (rsync -av --link-dest=...) to borg and having major speed issues. I set encryption to none and compression to none, but still borg needs 8 hours for a non-first backup, while rsync needs 6 minutes. It's about 350GB, mostly documents and images. The machine is quite slow, it's an ARM-based machine (odroid XU4), but still, rsync was able to do it pretty fast. Is there anything I can do, to get the speed to anything similar to what rsync was? A little bit slower would be okay, like an hour or maybe even two, but this is way to slow. Maybe some way of detecting the duplicates that is done differently? best regards Catscrash From tw at waldmann-edv.de Fri Apr 14 07:36:08 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 14 Apr 2017 13:36:08 +0200 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> Message-ID: <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> > I set encryption to none Good for speed in this special case. > and compression to none Not sure about this one, consider trying lz4 (little cpu load, less data to transfer/store). > but still borg needs 8 hours for a non-first backup That's strange for a non-first backup. borg usually detects most files are unchanged and skips them very fast. Only transfers if something has changed. The fast-skip detection depends on file mtime, file size and inode number. Check if these are stable for unchanged files. > while rsync needs 6 minutes. borg should be similarly fast for most files unchanged. For changed files, borg has a bit more to compute (plus optionally encryption and compression) than rsync but it will save you a lot of storage in some cases (like big files with little changes, like VM images). > It's about 350GB, mostly documents and images. The machine is quite > slow, it's an ARM-based machine (odroid XU4), but still, rsync was able > to do it pretty fast. borg 1.0 can only use 1 of the cores. borg 1.2 is planned to have multithreading. > Is there anything I can do, to get the speed to anything similar to what > rsync was? A little bit slower would be okay, like an hour or maybe even > two, but this is way to slow. Find out why fast-skip does not work. Is something touching the mtimes? Do you use a filesystem that does not have stable inode numbers? For the last case, there is a --ignore-inodes switch. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From ndbecker2 at gmail.com Fri Apr 14 08:12:17 2017 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 14 Apr 2017 12:12:17 +0000 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> Message-ID: I hope multi-threading is optional :) On Fri, Apr 14, 2017 at 7:36 AM Thomas Waldmann wrote: > > I set encryption to none > > Good for speed in this special case. > > > and compression to none > > Not sure about this one, consider trying lz4 (little cpu load, less data > to transfer/store). > > > but still borg needs 8 hours for a non-first backup > > That's strange for a non-first backup. > > borg usually detects most files are unchanged and skips them very fast. > Only transfers if something has changed. > > The fast-skip detection depends on file mtime, file size and inode > number. Check if these are stable for unchanged files. > > > while rsync needs 6 minutes. > > borg should be similarly fast for most files unchanged. > > For changed files, borg has a bit more to compute (plus optionally > encryption and compression) than rsync but it will save you a lot of > storage in some cases (like big files with little changes, like VM images). > > > It's about 350GB, mostly documents and images. The machine is quite > > slow, it's an ARM-based machine (odroid XU4), but still, rsync was able > > to do it pretty fast. > > borg 1.0 can only use 1 of the cores. > borg 1.2 is planned to have multithreading. > > > Is there anything I can do, to get the speed to anything similar to what > > rsync was? A little bit slower would be okay, like an hour or maybe even > > two, but this is way to slow. > > Find out why fast-skip does not work. Is something touching the mtimes? > Do you use a filesystem that does not have stable inode numbers? For the > last case, there is a --ignore-inodes switch. > > > > -- > > GPG ID: 9F88FB52FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From public at enkore.de Fri Apr 14 08:14:26 2017 From: public at enkore.de (Marian Beermann) Date: Fri, 14 Apr 2017 14:14:26 +0200 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> Message-ID: <3b88cf60-df6f-e3f1-5249-0220369e0fe4@enkore.de> taskset(1) On 14.04.2017 14:12, Neal Becker wrote: > I hope multi-threading is optional :) > > On Fri, Apr 14, 2017 at 7:36 AM Thomas Waldmann > wrote: > > > I set encryption to none > > Good for speed in this special case. > > > and compression to none > > Not sure about this one, consider trying lz4 (little cpu load, less data > to transfer/store). > > > but still borg needs 8 hours for a non-first backup > > That's strange for a non-first backup. > > borg usually detects most files are unchanged and skips them very fast. > Only transfers if something has changed. > > The fast-skip detection depends on file mtime, file size and inode > number. Check if these are stable for unchanged files. > > > while rsync needs 6 minutes. > > borg should be similarly fast for most files unchanged. > > For changed files, borg has a bit more to compute (plus optionally > encryption and compression) than rsync but it will save you a lot of > storage in some cases (like big files with little changes, like VM > images). > > > It's about 350GB, mostly documents and images. The machine is quite > > slow, it's an ARM-based machine (odroid XU4), but still, rsync was > able > > to do it pretty fast. > > borg 1.0 can only use 1 of the cores. > borg 1.2 is planned to have multithreading. > > > Is there anything I can do, to get the speed to anything similar > to what > > rsync was? A little bit slower would be okay, like an hour or > maybe even > > two, but this is way to slow. > > Find out why fast-skip does not work. Is something touching the mtimes? > Do you use a filesystem that does not have stable inode numbers? For the > last case, there is a --ignore-inodes switch. > > > > -- > > GPG ID: 9F88FB52FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From catscrash at catscrash.de Fri Apr 14 11:06:54 2017 From: catscrash at catscrash.de (Catscrash) Date: Fri, 14 Apr 2017 17:06:54 +0200 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> Message-ID: Hi, thanks for your answer, >> and compression to none > Not sure about this one, consider trying lz4 (little cpu load, less data > to transfer/store). That leads to 14 hours on a non-first backup run, I did try that already before and ended up starting from scratch with uncompressed to try if that's faste > The fast-skip detection depends on file mtime, file size and inode > number. Check if these are stable for unchanged files. > [...] > borg should be similarly fast for most files unchanged. > > For changed files, borg has a bit more to compute (plus optionally > encryption and compression) than rsync but it will save you a lot of > storage in some cases (like big files with little changes, like VM images). In the end it's only 67,22MB that's new, so it does recognize the files as unchanged: Time (start): Fri, 2017-04-14 04:00:13 Time (end): Fri, 2017-04-14 12:17:43 Duration: 8 hours 17 minutes 30.63 seconds Number of files: 258614 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 335.34 GB 335.34 GB 67.22 MB All archives: 670.68 GB 670.68 GB 304.49 GB Unique chunks Total chunks Chunk index: 311269 744901 The largest file system taking up the most space is this one /dev/mapper/LVM-Bilder on /srv/Bilder type ext4 (rw,relatime,data=ordered) The files are not touched or changed, mtime is for most several years ago, file size doesn't change as well and if inode would change, rsync would not be able to hardlink, since it's depending on inode as far as I know. Atime on the other hand is of course every night when the backup runs, but that shouldn't be an issue, right? > borg 1.0 can only use 1 of the cores. > borg 1.2 is planned to have multithreading. Okay sure, that might be an issue, but still the difference seems a bit huge for it just to be due to the cores used, right? > Find out why fast-skip does not work. Is something touching the mtimes? > Do you use a filesystem that does not have stable inode numbers? For the > last case, there is a --ignore-inodes switch. How do I know if it's fast-skipping or not? mtime is not changed, ext4 should have stable inode numbers. Nevertheless, I'm trying with the --ignore-inodes switch now and will let you know Thanks for your time! From catscrash at catscrash.de Fri Apr 14 11:15:44 2017 From: catscrash at catscrash.de (Catscrash) Date: Fri, 14 Apr 2017 17:15:44 +0200 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> Message-ID: <54760130-8429-2d51-b5fd-157dca1b5936@catscrash.de> > How do I know if it's fast-skipping or not? mtime is not changed, ext4 > should have stable inode numbers. Nevertheless, I'm trying with the > --ignore-inodes switch now and will let you know Wow, -- ignore-inode did the trick Archive fingerprint: 935e503e25900163a5ad922f890f590fc4a0f98272755673b671005d52340738 Time (start): Fri, 2017-04-14 17:08:44 Time (end): Fri, 2017-04-14 17:13:29 Duration: 4 minutes 45.03 seconds Number of files: 258616 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 335.34 GB 335.34 GB 30.75 MB All archives: 1.01 TB 1.01 TB 304.52 GB Unique chunks Total chunks Chunk index: 311916 1117275 ------------------------------------------------------------------------------ Ende der Sicherung Fri Apr 14 17:13:55 CEST 2017. Dauer: 317 Sekunden Do you have an idea for a reason for this? I thought ext4 has stable inodes? Is there a disadvantage to the ignore-inode switch? best regards From public at enkore.de Fri Apr 14 11:51:05 2017 From: public at enkore.de (Marian Beermann) Date: Fri, 14 Apr 2017 17:51:05 +0200 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: <54760130-8429-2d51-b5fd-157dca1b5936@catscrash.de> References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> <54760130-8429-2d51-b5fd-157dca1b5936@catscrash.de> Message-ID: <1c1a2427-e8ed-8008-ebc9-10c6f3ee72fd@enkore.de> On 14.04.2017 17:15, Catscrash wrote: >> How do I know if it's fast-skipping or not? mtime is not changed, ext4 >> should have stable inode numbers. Nevertheless, I'm trying with the >> --ignore-inodes switch now and will let you know > Wow, -- ignore-inode did the trick > You are accessing the filesystem directly, yeah (not via sshfs / NFS or something like that)? Cheers, Marian From catscrash at catscrash.de Fri Apr 14 14:27:31 2017 From: catscrash at catscrash.de (Catscrash) Date: Fri, 14 Apr 2017 20:27:31 +0200 Subject: [Borgbackup] Speed issues compared with rsync In-Reply-To: <1c1a2427-e8ed-8008-ebc9-10c6f3ee72fd@enkore.de> References: <560bd06c-3233-19db-ccd6-c501208a9182@catscrash.de> <162c8d33-5713-add6-d9ac-073edfcfa1d0@waldmann-edv.de> <54760130-8429-2d51-b5fd-157dca1b5936@catscrash.de> <1c1a2427-e8ed-8008-ebc9-10c6f3ee72fd@enkore.de> Message-ID: <5b698faf-613c-27b7-e085-66a4e60d6c0b@catscrash.de> Am 14.04.2017 um 17:51 schrieb Marian Beermann: > On 14.04.2017 17:15, Catscrash wrote: >>> How do I know if it's fast-skipping or not? mtime is not changed, ext4 >>> should have stable inode numbers. Nevertheless, I'm trying with the >>> --ignore-inodes switch now and will let you know >> Wow, -- ignore-inode did the trick >> > You are accessing the filesystem directly, yeah (not via sshfs / NFS or > something like that)? Yep. Direct access. Weirdly after it ran with the inode-option fast once it's now fast without the option as well... I'm very confused, but happy it works fine now, so I think everything is good... thanks for the help! From arthur at artifacial.org Sun Apr 23 14:50:05 2017 From: arthur at artifacial.org (Arthur Elsenaar) Date: Sun, 23 Apr 2017 20:50:05 +0200 Subject: [Borgbackup] borg upgrade cannot find repo Message-ID: Hi, Newbie here. I created a repo that I at first created on a local server. I then moved the drive offsite. In this case the repo part has changed because it runs on another system. When I try to borg upgrade repo, it errors out with Repository repo does not exist. Do I perhaps need to tell the repo has changed? Thanks, Arthur -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 801 bytes Desc: Message signed with OpenPGP URL: From tw at waldmann-edv.de Sun Apr 23 15:41:05 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 23 Apr 2017 21:41:05 +0200 Subject: [Borgbackup] borg upgrade cannot find repo In-Reply-To: References: Message-ID: <442785ac-8dfe-7f05-219a-27337263a096@waldmann-edv.de> > Newbie here. I created a repo that I at first created on a local server. I then moved the drive offsite. In this case the repo part has changed because it runs on another system. > > When I try to borg upgrade repo, it errors out with Repository repo does not exist. borg upgrade does not work for remote repos, we have a ticket about that on the issue tracker. But if you just want to continue using that repo in a similar way as before, you do not need to upgrade, right? -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From p.kolodziej at gmail.com Thu Apr 27 17:32:57 2017 From: p.kolodziej at gmail.com (=?UTF-8?B?UGF3ZcWCIEtvxYJvZHppZWo=?=) Date: Thu, 27 Apr 2017 23:32:57 +0200 Subject: [Borgbackup] problem with --patterns-from Message-ID: <615bdeec-5c3e-ba01-938c-e3b74b7bce37@gmail.com> Hi, I try to backup only .odt files using --paterns-from. My simlpe test case is: - directory /tmp/test with files f1.txt and f2.odt: $ mkdir /tmp/test $ date > /tmp/test/f1.txt $ date > /tmp/test/f2.odt - file p.txt: $ cat p.txt R /tmp/test + re:odt - /tmp/test then: $ borg init -e none /tmp/repo $ borg create --patterns-from p.txt /tmp/repo::{now} --list x /tmp/test so f2.odt was not stored and I don't understand why. It matches first pattern (re:odt) so it should be included. $ borg --version borg 1.1.0b4 I'm looking forward for any help. Best, Pawel From public at enkore.de Fri Apr 28 04:30:11 2017 From: public at enkore.de (Marian Beermann) Date: Fri, 28 Apr 2017 10:30:11 +0200 Subject: [Borgbackup] problem with --patterns-from In-Reply-To: <615bdeec-5c3e-ba01-938c-e3b74b7bce37@gmail.com> References: <615bdeec-5c3e-ba01-938c-e3b74b7bce37@gmail.com> Message-ID: Hi Pawel, you're not doing anything wrong, this is/was an issue in the patterns implementation. See https://github.com/borgbackup/borg/issues/2314 Cheers, Marian On 27.04.2017 23:32, Pawe? Ko?odziej wrote: > Hi, > I try to backup only .odt files using --paterns-from. > > My simlpe test case is: > - directory /tmp/test with files f1.txt and f2.odt: > $ mkdir /tmp/test > $ date > /tmp/test/f1.txt > $ date > /tmp/test/f2.odt > - file p.txt: > $ cat p.txt > R /tmp/test > + re:odt > - /tmp/test > > then: > $ borg init -e none /tmp/repo > $ borg create --patterns-from p.txt /tmp/repo::{now} --list > x /tmp/test > > so f2.odt was not stored and I don't understand why. It matches first > pattern (re:odt) so it should be included. > > $ borg --version > borg 1.1.0b4 > > I'm looking forward for any help. > > Best, > Pawel > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Sun Apr 30 08:50:52 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 30 Apr 2017 14:50:52 +0200 Subject: [Borgbackup] borgbackup beta 1.1.0b5 released Message-ID: https://github.com/borgbackup/borg/releases/tag/1.1.0b5 More details: see URL above. Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From jdc at uwo.ca Mon May 1 09:15:59 2017 From: jdc at uwo.ca (Dan Christensen) Date: Mon, 1 May 2017 09:15:59 -0400 Subject: [Borgbackup] borgbackup beta 1.1.0b5 released In-Reply-To: (Thomas Waldmann's message of "Sun, 30 Apr 2017 14:50:52 +0200") References: Message-ID: <87bmrcd8fk.fsf@uwo.ca> I use borg create --exclude-from=filename to exclude /run, /proc and /sys from my backups. With 1.1.0b5, I get errors like /run/user/1000: stat: [Errno 13] Permission denied: '/run/user/1000/gvfs' /proc/29566/task/29566/fd: stat: [Errno 2] No such file or directory: '/proc/29566/task/29566/fd/8' I assume this is because of the changes related to recursively searching an excluded subdirectory, but I had thought that that would only affect efficiency, not cause borg to give errors and exit with an error code. Two questions: 1) Can borg handle this more cleanly? 2) With 1.1.0b5, how would I change my exclude-from file to make this go away? Thanks, Dan From public at enkore.de Mon May 1 10:36:02 2017 From: public at enkore.de (Marian Beermann) Date: Mon, 1 May 2017 16:36:02 +0200 Subject: [Borgbackup] borgbackup beta 1.1.0b5 released In-Reply-To: <87bmrcd8fk.fsf@uwo.ca> References: <87bmrcd8fk.fsf@uwo.ca> Message-ID: <021d0317-0786-3256-fd55-d28cde1340f2@enkore.de> On 01.05.2017 15:15, Dan Christensen wrote: > I use > > borg create --exclude-from=filename > > to exclude /run, /proc and /sys from my backups. With 1.1.0b5, I get > errors like > > /run/user/1000: stat: [Errno 13] Permission denied: '/run/user/1000/gvfs' > /proc/29566/task/29566/fd: stat: [Errno 2] No such file or directory: '/proc/29566/task/29566/fd/8' > > I assume this is because of the changes related to recursively searching > an excluded subdirectory, but I had thought that that would only affect > efficiency, not cause borg to give errors and exit with an error code. > > Two questions: > > 1) Can borg handle this more cleanly? > > 2) With 1.1.0b5, how would I change my exclude-from file to make this go > away? > > Thanks, > > Dan Hi Dan, this went completely wrong, sorry about that. --exclude semantics were not intended to be changed and will be fixed. Cheers, Marian From zsolt.ero at gmail.com Thu May 4 05:42:55 2017 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Thu, 4 May 2017 11:42:55 +0200 Subject: [Borgbackup] how to back up 300 million files Message-ID: Hi, I am trying to solve the problem of backing up 300 million files, preferably with borg. The files are small, totalling only 100 GB all together (12 kb on average). Is it possible to use borg to handle such task? The documentation says it is quite limited in the number of files. I am looking for a solution which doesn't need to be changed even if the files grow to 10x the size of this. Preferably I'd like to back up over to a remote server, running "borg serve", having a local disk. I am OK to use 1.10 beta as long as the archives stay compatible with the final 1.10 version. Is that the case? Zsolt From zsolt.ero at gmail.com Thu May 4 06:20:23 2017 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Thu, 4 May 2017 12:20:23 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: References: Message-ID: Also, is this page still not updated to reflect 1.10 changes? http://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#note-about-archive-limitations Is MAX_OBJECT_SIZE a constant or can be set using run time parameters? On 4 May 2017 at 11:42, Zsolt Ero wrote: > Hi, > > I am trying to solve the problem of backing up 300 million files, > preferably with borg. The files are small, totalling only 100 GB all > together (12 kb on average). > > Is it possible to use borg to handle such task? The documentation says > it is quite limited in the number of files. I am looking for a > solution which doesn't need to be changed even if the files grow to > 10x the size of this. > > Preferably I'd like to back up over to a remote server, running "borg > serve", having a local disk. > > I am OK to use 1.10 beta as long as the archives stay compatible with > the final 1.10 version. Is that the case? > > Zsolt From public at enkore.de Thu May 4 06:26:36 2017 From: public at enkore.de (Marian Beermann) Date: Thu, 4 May 2017 12:26:36 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: References: Message-ID: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> As far as I can see the information there is correct and complete. MAX_OBJECT_SIZE is an internal constant > ... limited in size to MAX_OBJECT_SIZE (20MiB). Regarding 1.1.x beta compatibility with stable releases; there is no intent to break it. Doing so would make them pointless, since no one would test such an unstable release. Cheers, Marian On 04.05.2017 12:20, Zsolt Ero wrote: > Also, is this page still not updated to reflect 1.10 changes? > http://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#note-about-archive-limitations > > Is MAX_OBJECT_SIZE a constant or can be set using run time parameters? > > On 4 May 2017 at 11:42, Zsolt Ero wrote: >> Hi, >> >> I am trying to solve the problem of backing up 300 million files, >> preferably with borg. The files are small, totalling only 100 GB all >> together (12 kb on average). >> >> Is it possible to use borg to handle such task? The documentation says >> it is quite limited in the number of files. I am looking for a >> solution which doesn't need to be changed even if the files grow to >> 10x the size of this. >> >> Preferably I'd like to back up over to a remote server, running "borg >> serve", having a local disk. >> >> I am OK to use 1.10 beta as long as the archives stay compatible with >> the final 1.10 version. Is that the case? >> >> Zsolt > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From maurice.libes at osupytheas.fr Thu May 4 06:47:22 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 4 May 2017 12:47:22 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> Message-ID: Le 04/05/2017 ? 12:26, Marian Beermann a ?crit : > As far as I can see the information there is correct and complete. > MAX_OBJECT_SIZE is an internal constant > >> ... limited in size to MAX_OBJECT_SIZE (20MiB). > Regarding 1.1.x beta compatibility with stable releases; there is no > intent to break it. Doing so would make them pointless, since no one > would test such an unstable release. > > Cheers, Marian > > On 04.05.2017 12:20, Zsolt Ero wrote: >> Also, is this page still not updated to reflect 1.10 changes? >> http://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#note-about-archive-limitations >> >> Is MAX_OBJECT_SIZE a constant or can be set using run time parameters? >> >> On 4 May 2017 at 11:42, Zsolt Ero wrote: >>> Hi, >>> >>> I am trying to solve the problem of backing up 300 million files, >>> preferably with borg. The files are small, totalling only 100 GB all >>> together (12 kb on average).` other answer/question from another point of view of a neophyte: Is borg an appropriate solution in this case of very small files (12kb) , since borg will never split the files into chunks? so don't we lose the benefit of deduplication ? or am I wrong? I don't remember what is the limit for a file to be split into chunks ML >>> >>> Is it possible to use borg to handle such task? The documentation says >>> it is quite limited in the number of files. I am looking for a >>> solution which doesn't need to be changed even if the files grow to >>> 10x the size of this. >>> >>> Preferably I'd like to back up over to a remote server, running "borg >>> serve", having a local disk. >>> >>> I am OK to use 1.10 beta as long as the archives stay compatible with >>> the final 1.10 version. Is that the case? >>> >>> Zsolt >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: maurice_libes.vcf Type: text/x-vcard Size: 176 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2971 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Thu May 4 06:59:07 2017 From: public at enkore.de (Marian Beermann) Date: Thu, 4 May 2017 12:59:07 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> Message-ID: <241938b7-c4dc-358f-895d-3b399cedce89@enkore.de> On 04.05.2017 12:47, Maurice Libes wrote: > other answer/question from another point of view of a neophyte: > > Is borg an appropriate solution in this case of very small files (12kb) > , since borg will never split the files into chunks? > so don't we lose the benefit of deduplication ? > or am I wrong? > I don't remember what is the limit for a file to be split into chunks > Small files won't be split into chunks, but they will still be deduplicated. However, only one bit in a file needs to change and borg will store the whole ~12kb. Whether this is a problem is application-dependent, i.e. are files actually changed. If files are not modified, it's not an issue. Cheers, Marian From maurice.libes at osupytheas.fr Thu May 4 07:08:48 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 4 May 2017 13:08:48 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: <241938b7-c4dc-358f-895d-3b399cedce89@enkore.de> References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> <241938b7-c4dc-358f-895d-3b399cedce89@enkore.de> Message-ID: <578a2753-617c-868a-bb68-629aaf051cc0@osupytheas.fr> Le 04/05/2017 ? 12:59, Marian Beermann a ?crit : > On 04.05.2017 12:47, Maurice Libes wrote: >> other answer/question from another point of view of a neophyte: >> >> Is borg an appropriate solution in this case of very small files (12kb) >> , since borg will never split the files into chunks? >> so don't we lose the benefit of deduplication ? >> or am I wrong? >> I don't remember what is the limit for a file to be split into chunks >> > Small files won't be split into chunks, but they will still be > deduplicated. If they are not split, so you talk about deduplication at the file "level", if I understand? like another incremental backup which do not backup again the same files which are already presents? (let's say it is not what I call deduplication, for me deduplication is a process at the block file level, but may be it is a question of terms) ML > However, only one bit in a file needs to change and borg > will store the whole ~12kb. Whether this is a problem is > application-dependent, i.e. are files actually changed. If files are not > modified, it's not an issue. > > Cheers, Marian > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: maurice_libes.vcf Type: text/x-vcard Size: 176 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2971 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Thu May 4 07:12:42 2017 From: public at enkore.de (Marian Beermann) Date: Thu, 4 May 2017 13:12:42 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: <578a2753-617c-868a-bb68-629aaf051cc0@osupytheas.fr> References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> <241938b7-c4dc-358f-895d-3b399cedce89@enkore.de> <578a2753-617c-868a-bb68-629aaf051cc0@osupytheas.fr> Message-ID: On 04.05.2017 13:08, Maurice Libes wrote: > Le 04/05/2017 ? 12:59, Marian Beermann a ?crit : >> On 04.05.2017 12:47, Maurice Libes wrote: >>> other answer/question from another point of view of a neophyte: >>> >>> Is borg an appropriate solution in this case of very small files (12kb) >>> , since borg will never split the files into chunks? >>> so don't we lose the benefit of deduplication ? >>> or am I wrong? >>> I don't remember what is the limit for a file to be split into chunks >>> >> Small files won't be split into chunks, but they will still be >> deduplicated. > If they are not split, so you talk about deduplication at the file > "level", if I understand? Yes > like another incremental backup which do not backup again the same > files which are already presents? Depends - simple incremental backup (e.g. rsnapshot) yes, but diff-based incremental backups would only store a diff. > (let's say it is not what I call deduplication, > for me deduplication is a process at the block file level, but may be it > is a question of terms) > > ML The two cases are the same if a file consists only of one "block" ;) Cheers, Marian From mario at emmenlauer.de Thu May 4 06:59:02 2017 From: mario at emmenlauer.de (Mario Emmenlauer) Date: Thu, 4 May 2017 12:59:02 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> Message-ID: <6bdda1ab-75d3-d051-bfde-91cd0de5945e@emmenlauer.de> On 04.05.2017 12:47, Maurice Libes wrote: > Le 04/05/2017 ? 12:26, Marian Beermann a ?crit : >> As far as I can see the information there is correct and complete. >> MAX_OBJECT_SIZE is an internal constant >> >>> ... limited in size to MAX_OBJECT_SIZE (20MiB). >> Regarding 1.1.x beta compatibility with stable releases; there is no >> intent to break it. Doing so would make them pointless, since no one >> would test such an unstable release. >> >> Cheers, Marian >> >> On 04.05.2017 12:20, Zsolt Ero wrote: >>> Also, is this page still not updated to reflect 1.10 changes? >>> http://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#note-about-archive-limitations >>> >>> Is MAX_OBJECT_SIZE a constant or can be set using run time parameters? >>> >>> On 4 May 2017 at 11:42, Zsolt Ero wrote: >>>> Hi, >>>> >>>> I am trying to solve the problem of backing up 300 million files, >>>> preferably with borg. The files are small, totalling only 100 GB all >>>> together (12 kb on average).` > other answer/question from another point of view of a neophyte: > > Is borg an appropriate solution in this case of very small files (12kb) , since borg will never split the files into chunks? > so don't we lose the benefit of deduplication ? > or am I wrong? > I don't remember what is the limit for a file to be split into chunks If some of the files are identical, they would still be de-duplicated. But I agree that its not the standard use case of borg. Zsolt, do you have many duplicate files in your collection? If not, do the files often change? Did you think about a simpler backup solution like rsync with hard-links? Just my two cents. Cheers, Mario From zsolt.ero at gmail.com Thu May 4 07:22:38 2017 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Thu, 4 May 2017 13:22:38 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: <6bdda1ab-75d3-d051-bfde-91cd0de5945e@emmenlauer.de> References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> <6bdda1ab-75d3-d051-bfde-91cd0de5945e@emmenlauer.de> Message-ID: The files are not identical, and are not compressible (tiny jpg files for a map-based viewer), and they will never change. I think I'll write a custom script to tar them up into 500 tar files based on subdirs and just store those tars somewhere cheap (S3, etc.). Zsolt On 4 May 2017 at 12:59, Mario Emmenlauer wrote: > > On 04.05.2017 12:47, Maurice Libes wrote: >> Le 04/05/2017 ? 12:26, Marian Beermann a ?crit : >>> As far as I can see the information there is correct and complete. >>> MAX_OBJECT_SIZE is an internal constant >>> >>>> ... limited in size to MAX_OBJECT_SIZE (20MiB). >>> Regarding 1.1.x beta compatibility with stable releases; there is no >>> intent to break it. Doing so would make them pointless, since no one >>> would test such an unstable release. >>> >>> Cheers, Marian >>> >>> On 04.05.2017 12:20, Zsolt Ero wrote: >>>> Also, is this page still not updated to reflect 1.10 changes? >>>> http://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#note-about-archive-limitations >>>> >>>> Is MAX_OBJECT_SIZE a constant or can be set using run time parameters? >>>> >>>> On 4 May 2017 at 11:42, Zsolt Ero wrote: >>>>> Hi, >>>>> >>>>> I am trying to solve the problem of backing up 300 million files, >>>>> preferably with borg. The files are small, totalling only 100 GB all >>>>> together (12 kb on average).` >> other answer/question from another point of view of a neophyte: >> >> Is borg an appropriate solution in this case of very small files (12kb) , since borg will never split the files into chunks? >> so don't we lose the benefit of deduplication ? >> or am I wrong? >> I don't remember what is the limit for a file to be split into chunks > > If some of the files are identical, they would still be de-duplicated. > But I agree that its not the standard use case of borg. > > Zsolt, do you have many duplicate files in your collection? If not, do > the files often change? Did you think about a simpler backup solution > like rsync with hard-links? > > Just my two cents. > > Cheers, > > Mario > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From maurice.libes at osupytheas.fr Thu May 4 07:44:34 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 4 May 2017 13:44:34 +0200 Subject: [Borgbackup] borg basic understanding Message-ID: <6a9f17b9-5221-3b6e-48c7-915b896e4c72@osupytheas.fr> Hi to all for that matter, In the aim of making a presentation and slides of Borgbackup to my sysadmin collegues may I have some informations about some simple internals processes of borg? to improve my understanding , I try to details what are the main actions made on both client and serveur sides To my understanding may we say that the main actions of backup processes are made on the client side? or no ? from client side : - borg parses each files to backup - checks if the file have been touched or modified - If the file is new or has been modified : - borg splits the file into several chunks - then checks in the local cache (on client side) if a chunk already "exists" (has been already backuped) or not? - If the chunk is new (never seen before) : - the chunk is compressed and then encrypted *before* sending it over the network - sends it over the network to the server [main features of the backup processe relies on the local cache no?] on server side : what are the main actions in the backup process on borg server side ? except to put the received chunks in the appropriate archive and manage the repositories ? thanks if you can correct me or add some more informations or appropriate links sorry if i am too far of the reality and complexity of the process ML if interested i can give my presentation for french audience -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: maurice_libes.vcf Type: text/x-vcard Size: 176 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2971 bytes Desc: Signature cryptographique S/MIME URL: From tw at waldmann-edv.de Thu May 4 08:55:44 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 4 May 2017 14:55:44 +0200 Subject: [Borgbackup] borg basic understanding In-Reply-To: <6a9f17b9-5221-3b6e-48c7-915b896e4c72@osupytheas.fr> References: <6a9f17b9-5221-3b6e-48c7-915b896e4c72@osupytheas.fr> Message-ID: <1c23dbff-0b3b-0fe5-b3c4-ef37edb0184b@waldmann-edv.de> > from client side : > > - borg parses each files to backup maybe say "discovers" not "parses". > - checks if the file have been touched or modified yes, mtime, inode number, size is checked. > - If the file is new or has been modified : > - borg splits the file into several chunks yes, except if the file is rather small and thus only results in 1 chunk. important here is that it is "content defined" chunking, so the cutting offsets are not fixed raster, but determined by the content (the rolling hash value). so insertions and deletions in a file do not trigger a lot new chunks just because of the shift. > - then checks in the local cache (on client side) if a chunk already > "exists" (has been already backuped) or not? > > - If the chunk is new (never seen before) : > - the chunk is compressed and then encrypted *before* sending it over > the network after encryption, the data is additionally authenticated (a MAC is computed) to "protect" it against tampering. so it is "authenticated encryption". > - sends it over the network to the server > > [main features of the backup processe relies on the local cache no?] yes, the local chunks cache and files cache are responsible for the good speed. there are some more little details on the client side when encryption is on (default): - the chunk IDs are MACs (not just hashes over the plaintext as that would tell too much about your data). only who can use the key can compute these MACs. - the chunker is seeded with a fixed, but per-key random value (so the cutting offsets and lengths of cut pieces differ from another repo / another key chunking the same data). only who can use the key can cut chunks in that specific way. this is both to protect privacy / confidentiality of your data and counter attacks on untrusted repo storage. > on server side : > what are the main actions in the backup process on borg server side ? it's basically a key/value store (with locking and an internal index). it also computes CRC32 checksums over the stored entries, so a repo server side check for accidential corruption can be done. as it only sees encrypted data and metadata, it can not (and shall not) do any high-level operations. > if interested i can give my presentation for french audience if you publish the slides somewhere, we could link to them. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From zsolt.ero at gmail.com Thu May 4 09:18:16 2017 From: zsolt.ero at gmail.com (Zsolt Ero) Date: Thu, 4 May 2017 15:18:16 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> <6bdda1ab-75d3-d051-bfde-91cd0de5945e@emmenlauer.de> Message-ID: I can confirm that it is not possible to create the initial backup even in 1.10: borg create --stats --progress --one-file-system backup::2 imagetiles Killed stale lock production at 207216302901725.27788-0. Removed stale exclusive roster lock for pid 27788. Removed stale exclusive roster lock for pid 27788. Killed stale lock production at 207216302901725.27788-0. Removed stale exclusive roster lock for pid 27788. Removed stale exclusive roster lock for pid 27788. KilledGB O 53.09 GB C 310.07 kB D 4259789 N imagetiles/3502f5816ea3ac8b/TileGroup205/9-173-99.jpg borg info -v backup Killed stale lock production at 207216302901725.28349-0. Removed stale exclusive roster lock for pid 28349. Killed stale lock production at 207216302901725.28349-0. Removed stale exclusive roster lock for pid 28349. Removed stale exclusive roster lock for pid 28349. Repository ID: e7ae835fb34055a6d6f1d2ef469a681eeb65e78a5e2074bd24edef7d975d1462 Location: ... Encrypted: Yes (authenticated BLAKE2b) Cache: /root/.cache/borg/e7ae835fb34055a6d6f1d2ef469a681eeb65e78a5e2074bd24edef7d975d1462 Security dir: /root/.config/borg/security/e7ae835fb34055a6d6f1d2ef469a681eeb65e78a5e2074bd24edef7d975d1462 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size All archives: 57.76 GB 56.09 GB 54.83 GB Unique chunks Total chunks Chunk index: 3815976 4460576 On 4 May 2017 at 13:22, Zsolt Ero wrote: > The files are not identical, and are not compressible (tiny jpg files > for a map-based viewer), and they will never change. > > I think I'll write a custom script to tar them up into 500 tar files > based on subdirs and just > store those tars somewhere cheap (S3, etc.). > > Zsolt > > > > On 4 May 2017 at 12:59, Mario Emmenlauer wrote: >> >> On 04.05.2017 12:47, Maurice Libes wrote: >>> Le 04/05/2017 ? 12:26, Marian Beermann a ?crit : >>>> As far as I can see the information there is correct and complete. >>>> MAX_OBJECT_SIZE is an internal constant >>>> >>>>> ... limited in size to MAX_OBJECT_SIZE (20MiB). >>>> Regarding 1.1.x beta compatibility with stable releases; there is no >>>> intent to break it. Doing so would make them pointless, since no one >>>> would test such an unstable release. >>>> >>>> Cheers, Marian >>>> >>>> On 04.05.2017 12:20, Zsolt Ero wrote: >>>>> Also, is this page still not updated to reflect 1.10 changes? >>>>> http://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#note-about-archive-limitations >>>>> >>>>> Is MAX_OBJECT_SIZE a constant or can be set using run time parameters? >>>>> >>>>> On 4 May 2017 at 11:42, Zsolt Ero wrote: >>>>>> Hi, >>>>>> >>>>>> I am trying to solve the problem of backing up 300 million files, >>>>>> preferably with borg. The files are small, totalling only 100 GB all >>>>>> together (12 kb on average).` >>> other answer/question from another point of view of a neophyte: >>> >>> Is borg an appropriate solution in this case of very small files (12kb) , since borg will never split the files into chunks? >>> so don't we lose the benefit of deduplication ? >>> or am I wrong? >>> I don't remember what is the limit for a file to be split into chunks >> >> If some of the files are identical, they would still be de-duplicated. >> But I agree that its not the standard use case of borg. >> >> Zsolt, do you have many duplicate files in your collection? If not, do >> the files often change? Did you think about a simpler backup solution >> like rsync with hard-links? >> >> Just my two cents. >> >> Cheers, >> >> Mario >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Thu May 4 12:21:11 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 4 May 2017 18:21:11 +0200 Subject: [Borgbackup] how to back up 300 million files In-Reply-To: References: <97b63479-6e2f-f9e3-e5b7-4a3c341519cf@enkore.de> <6bdda1ab-75d3-d051-bfde-91cd0de5945e@emmenlauer.de> Message-ID: On 05/04/2017 03:18 PM, Zsolt Ero wrote: > I can confirm that it is not possible to create the initial backup even in 1.10: More infos about that would be nice. > borg create --stats --progress --one-file-system backup::2 imagetiles > Killed stale lock production at 207216302901725.27788-0. > Removed stale exclusive roster lock for pid 27788. > Removed stale exclusive roster lock for pid 27788. > Killed stale lock production at 207216302901725.27788-0. > Removed stale exclusive roster lock for pid 27788. > Removed stale exclusive roster lock for pid 27788. > KilledGB O 53.09 GB C 310.07 kB D 4259789 N > imagetiles/3502f5816ea3ac8b/TileGroup205/9-173-99.jpg The "killed stale lock" msg itself is harmless. But the existance of a stale lock somehow points to some previous issue/crash/killing. So, did you see tracebacks before that and can we have them please? > borg info -v backup I think you need to call it with repo::archive to see how much of the maximum archive root chunk size was used for that archive. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From maurice.libes at osupytheas.fr Fri Jun 2 06:24:18 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Fri, 2 Jun 2017 12:24:18 +0200 Subject: [Borgbackup] borg on windows? Message-ID: hi to all don't know if it is a faq, but, is there any chance to see borg running on a windows system (client side at least) ? as it is written in python is there a known problem of development or integration under Windows? I am not a Windows fan but in term on park management, it would be interesting to rely on borg backup? thanks ML -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Fri Jun 2 06:36:47 2017 From: public at enkore.de (Marian Beermann) Date: Fri, 2 Jun 2017 12:36:47 +0200 Subject: [Borgbackup] borg on windows? In-Reply-To: References: Message-ID: There are multiple ways to use Borg on Windows. - Experimental: The "windows" branch (has not been merged with the main development branch in a while). This is not really stable and sometimes crashes, also no binaries available. Some advanced features (like ACLs and ADS) are already supported. - Via Cygwin; there is an unofficial installer including Cygwin - Via Windows 10 WSL; there is a ticket where people tried it and it works. The latter two can't and never will support things like ACLs, ADS, EA, OIDs, ... and so on. See tickets with "os: windows" label: https://github.com/borgbackup/borg/issues?q=is%3Aissue+is%3Aopen+label%3A%22os%3A+windows%22 Cheers, Marian On 02.06.2017 12:24, Maurice Libes wrote: > hi to all > > don't know if it is a faq, but, > > is there any chance to see borg running on a windows system (client > side at least) ? > > as it is written in python is there a known problem of development or > integration under Windows? > > I am not a Windows fan but in term on park management, it would be > interesting to rely on borg backup? > > thanks > > ML > > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From levin at straubis.org Fri Jun 2 06:35:43 2017 From: levin at straubis.org (Levin) Date: Fri, 02 Jun 2017 12:35:43 +0200 Subject: [Borgbackup] borg on windows? In-Reply-To: References: Message-ID: <163FFA99-5BDE-4537-B91F-2DF6B79B8EDE@straubis.org> Hi, at the moment you are able to run borg on cygwin or on the Win10 Linux Subsystem (both experimental) See https://borgbackup.readthedocs.io/en/stable/installation.html#source-install. Levin Am 2. Juni 2017 12:24:18 MESZ schrieb Maurice Libes : >hi to all > >don't know if it is a faq, but, > >is there any chance to see borg running on a windows system (client >side at least) ? > >as it is written in python is there a known problem of development or >integration under Windows? > >I am not a Windows fan but in term on park management, it would be >interesting to rely on borg backup? > >thanks > >ML > > >-- >M. LIBES >Service Informatique OSU Pytheas - UMS 3470 CNRS >Batiment Oceanomed >Campus de Luminy >13288 Marseille cedex 9 >Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: From nerbrume at free.fr Sat Jun 10 07:54:04 2017 From: nerbrume at free.fr (nerbrume at free.fr) Date: Sat, 10 Jun 2017 13:54:04 +0200 (CEST) Subject: [Borgbackup] Windows troubles In-Reply-To: <1483879508.364255766.1497092790786.JavaMail.root@zimbra59-e10.priv.proxad.net> Message-ID: <887034222.364398531.1497095644089.JavaMail.root@zimbra59-e10.priv.proxad.net> Hi, I'm having some troubles with backups made from a remote windows box. Before submitting any bugs, I would be glad to have some feedback on what is expected/normal. First, my setup : I installed borg on windows 10 using billyc's installer (https://github.com/borgbackup/borg/issues/440#issuecomment-279491415)(so, cygwin), and borg is backing up on a remote linux box which also have borg. As the volume of data was huge, I first made two local backups from the linux box, using a usb drive. These backups seems fine. Later, I made several (~10) remote backup from the windows box, some of them being partial (.checkpoint): 2017-04-09 Sun, 2017-04-09 18:31:07 <- local 2017-04-12 Wed, 2017-04-12 20:02:29 <- local 2017-05-30.checkpoint Tue, 2017-05-30 20:39:30 <- remote 2017-06-01.checkpoint Thu, 2017-06-01 01:00:25 <- remote 2017-06-01.checkpoint.1 Thu, 2017-06-01 20:53:24 <- remote 2017-06-01.checkpoint.2 Thu, 2017-06-01 21:06:14 <- remote 2017-06-04 Sun, 2017-06-04 21:28:42 <- remote 2017-06-06 Tue, 2017-06-06 01:00:26 <- remote 2017-06-07 Wed, 2017-06-07 01:00:26 <- remote 2017-06-09 Fri, 2017-06-09 01:00:24 <- remote 2017-06-10 Sat, 2017-06-10 01:00:28 <- remote The two "local" backups and the last remote one (2017-06-10) have a original size of ~860GB Each of the "remote" backups had a dedup. size of 500-1000 MB, and a original size of ~940GB, due to the inclusion of a non-needed folder (80GB containing various globally unique data, which would, I guess, be at least 50GB after dedup+compression). 1) I deleted all but the last "remote" backups (wanting to reclaim the ~80GB lost). Before deletion: >borg info backupBorg::2017-06-10 Name: 2017-06-10 Fingerprint: d79c57fada25ab7d3e6bb07a0e642d7d28ffb704c333a56d814490c991b232d7 Hostname: UN Username: **** Time (start): Sat, 2017-06-10 01:00:28 Time (end): Sat, 2017-06-10 01:17:23 Command line: /usr/bin/borg create -sv -C lzma,5 --exclude-from borg//backupExclude ssh://****@****///backupBorg::{now:%Y-%m-%d} /BackupSource/ Number of files: 317235 Original size Compressed size Deduplicated size This archive: 859.60 GB 663.66 GB 736.28 MB All archives: 7.73 TB 6.07 TB 557.82 GB Unique chunks Total chunks Chunk index: 538363 5645695 And after deletion (omitting identical first lines): >borg info backupBorg::2017-06-10 Original size Compressed size Deduplicated size This archive: 859.60 GB 663.66 GB 77.26 GB All archives: 2.55 TB 1.99 TB 554.58 GB Unique chunks Total chunks Chunk index: 532222 1851377 Having reclaimed only 3GB seemed suspicious, so I ran a repair : >borg check --repair backupBorg 132 orphaned objects found! Archive consistency check complete, problems found. After that : >borg info backupBorg::2017-06-10 Original size Compressed size Deduplicated size This archive: 859.60 GB 663.66 GB 77.26 GB All archives: 2.55 TB 1.99 TB 554.63 GB Unique chunks Total chunks Chunk index: 531596 1850784 How can the All archive Dedup size be bigger !? And how come I did not recover the ~50-80GB from the non-needed folder ? Should I worry about the health of my repo ? 2) This might be related, but mounting the "remote" backups end up in some folders (but not all) being unreadable. ls throws : ls: error reading directory .: Input/output error (original in french: "ls: lecture du r?pertoire .: Erreur d'entr?e/sortie") while borg throws (having used borg mount -f) : Exception after kill: Traceback (most recent call last): File "handlers.pxi", line 433, in llfuse.capi.fuse_releasedir (src/llfuse/capi_linux.c:11578) File "handlers.pxi", line 434, in llfuse.capi.fuse_releasedir (src/llfuse/capi_linux.c:11533) File "/usr/lib/python3/dist-packages/llfuse/pyapi.py", line 462, in releasedir def releasedir(self, fh): File "/usr/lib/python3/dist-packages/borg/helpers.py", line 1290, in handler raise exc_cls borg.helpers.SigTerm Exception after kill: Traceback (most recent call last): File "handlers.pxi", line 406, in llfuse.capi.fuse_readdir (src/llfuse/capi_linux.c:11058) File "handlers.pxi", line 411, in llfuse.capi.fuse_readdir (src/llfuse/capi_linux.c:10962) File "misc.pxi", line 32, in llfuse.capi.fill_c_stat (src/llfuse/capi_linux.c:16219) OverflowError: can't convert negative value to uid_t (last 6 lines repeated 2 times) However, borg check does not return any error, and so did a borg extract --dry-run. I also hand-extracted some files (3-4), and everything is matching the original. Anything I might try to help devs investigate, or is windows considered "not supported" ? Again, should I worry about the health of my repo ? Thanks a bunch, N From giuseppe.arvati at gmail.com Wed Jun 14 12:14:15 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Wed, 14 Jun 2017 18:14:15 +0200 Subject: [Borgbackup] no dedup Message-ID: <98b07fbf-54c0-c15a-cb92-6aefdcf401d3@gmail.com> Hello I'm new to borgbackup and after some tests I'm not able do view deduplication. Let me explain. I want to backup a daily VM snapshot into a borg repository, now these VM are saved on a NAS and the name is something like VM01-xxx.xva where xxx is the day. I created 2 archive in the same repository the first with a VM and the second with the same VM of the next day. There are no big differences from the to VM. Command line: /usr/local/bin/borg create -C lz4 /mnt/vm.borg::traccar-22 /mnt/xen3/Traccar-22.xva Command line: /usr/local/bin/borg create -C lz4 /mnt/vm.borg::traccar-23 /mnt/xen3/Traccar-23.xva borg info /mnt/vm.borg::traccar-23 Name: traccar-23 Fingerprint: 627f648409506904f72869b7c2afa0c17ca1730e745f68dd6c303e4d9526804a Hostname: apamfs2 Username: root Time (start): Mon, 2017-06-12 15:12:38 Time (end): Mon, 2017-06-12 15:15:40 Command line: /usr/local/bin/borg create -C lz4 /mnt/vm.borg::traccar-23 /mnt/xen3/Traccar-23.xva Number of files: 1 Original size Compressed size Deduplicated size This archive: 7.45 GB 2.26 GB 2.23 GB All archives: 14.45 GB 4.43 GB 4.40 GB Unique chunks Total chunks Chunk index: 5420 5551 As you can see the archive traccar-23 is an half of all archive. no dedup I tried to rename the vm before create the archive, so they have the same name/path into the archive, following FAQ "it always chunks all my files, even unchanged ones" Using Attic in the same way I had these results ( same 2 files ) Command line: /usr/local/bin/attic create --stat /mnt/vm.attic::traccar-23 /mnt/xen3/Traccar-23.xva Number of files: 1 Original size Compressed size Deduplicated size This archive: 7.45 GB 1.78 GB 332.89 MB All archives: 14.46 GB 3.50 GB 1.96 GB So what's the problem ? I made the tests first with python 3.6.1 self compiled and then with CentOS python 3.4 Second problem. Borg always show me a warning message about pure python msgpack and slow performance I installed msgpack coming with CentoOs but nothing What I need to do to ? Thank you very much to all guys who want help me. Giuseppe From tw at waldmann-edv.de Wed Jun 14 22:29:47 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 15 Jun 2017 04:29:47 +0200 Subject: [Borgbackup] no dedup In-Reply-To: <98b07fbf-54c0-c15a-cb92-6aefdcf401d3@gmail.com> References: <98b07fbf-54c0-c15a-cb92-6aefdcf401d3@gmail.com> Message-ID: <31c52378-ff2f-70ea-9970-adf3f62d95b8@waldmann-edv.de> > I'm new to borgbackup... Welcome! > I tried to rename the vm before create the archive, so they have the > same name/path into the archive, Paths do not matter for dedup. Only content does. Maybe .xva is not a good file format for deduplication? > Using Attic in the same way I had these results Attic has a more fine granular deduplication. You can have that with borg, too (see --chunker-params), but if you do that, you'll have to live with the consequences: It will create as many chunks as attic did and will consume lots of resources to manage them. > Borg always show me a warning message about pure > python msgpack and slow performance I installed msgpack coming with > CentoOs but nothing Contact the centos msgpack package maintainer, the package obviously misses the compiled version of msgpack and thus it falls back to the slower pure-python version. > Thank you very much to all guys who want help me. You're welcome. If you want to help your helpers, try to hit enter more often so that not all your content ends up in one large line. Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Sun Jun 18 11:05:10 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 18 Jun 2017 17:05:10 +0200 Subject: [Borgbackup] borgbackup beta 1.1.0b6 released Message-ID: <01219d74-019d-133f-fa4e-a7a3e1c9001a@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.1.0b6 More details: see URL above. Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From public at enkore.de Sun Jun 18 11:07:13 2017 From: public at enkore.de (Marian Beermann) Date: Sun, 18 Jun 2017 17:07:13 +0200 Subject: [Borgbackup] borgbackup beta 1.1.0b6 released In-Reply-To: <01219d74-019d-133f-fa4e-a7a3e1c9001a@waldmann-edv.de> References: <01219d74-019d-133f-fa4e-a7a3e1c9001a@waldmann-edv.de> Message-ID: <90a24c1c-c10e-a0b9-314c-12b3077dfcf9@enkore.de> On 18.06.2017 17:05, Thomas Waldmann wrote: > https://github.com/borgbackup/borg/releases/tag/1.1.0b6 > > More details: see URL above. > > Cheers, > > Thomas > Note: This fixes the notorious --exclude-dumps-a-ton-of-warnings bug. From giuseppe.arvati at gmail.com Tue Jun 20 03:16:02 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Tue, 20 Jun 2017 09:16:02 +0200 Subject: [Borgbackup] no dedup In-Reply-To: <31c52378-ff2f-70ea-9970-adf3f62d95b8@waldmann-edv.de> References: <98b07fbf-54c0-c15a-cb92-6aefdcf401d3@gmail.com> <31c52378-ff2f-70ea-9970-adf3f62d95b8@waldmann-edv.de> Message-ID: Hello Thomas, thank you for your answer I'll try different setting of --chunker-params asap to discover which best fit my needs Sorry for missing "enter" but I hit enter a lot to keep mail readable but I think my email client thunderbird did some strange conversions Giuseppe Il 15/06/2017 04:29, Thomas Waldmann ha scritto: >> I'm new to borgbackup... > Welcome! > >> I tried to rename the vm before create the archive, so they have the >> same name/path into the archive, > Paths do not matter for dedup. Only content does. > > Maybe .xva is not a good file format for deduplication? > >> Using Attic in the same way I had these results > Attic has a more fine granular deduplication. > > You can have that with borg, too (see --chunker-params), but if you do > that, you'll have to live with the consequences: > > It will create as many chunks as attic did and will consume lots of > resources to manage them. > >> Borg always show me a warning message about pure >> python msgpack and slow performance I installed msgpack coming with >> CentoOs but nothing > Contact the centos msgpack package maintainer, the package obviously > misses the compiled version of msgpack and thus it falls back to the > slower pure-python version. > >> Thank you very much to all guys who want help me. > You're welcome. > > If you want to help your helpers, try to hit enter more often so that > not all your content ends up in one large line. > > Cheers, Thomas > From nerbrume at free.fr Thu Jun 22 03:29:50 2017 From: nerbrume at free.fr (nerbrume at free.fr) Date: Thu, 22 Jun 2017 09:29:50 +0200 (CEST) Subject: [Borgbackup] Windows troubles In-Reply-To: <887034222.364398531.1497095644089.JavaMail.root@zimbra59-e10.priv.proxad.net> Message-ID: <1933950861.413908274.1498116590689.JavaMail.root@zimbra59-e10.priv.proxad.net> Hi, Some update on my troubles. First, something I overlooked in my previous message, but might be important : the windows machine is using borg 1.0.10, the linux (repo) is using 1.0.9 (but this shouldn't be a problem, right ?) 1) About the strange size of my archives/repo --------------------------------------------- I extracted an archive (2017-06-13), and recreated it, on another repo. Then, I deleted all the archives but this one on the original repo: > borg list backupBorg <-- original 2017-06-13 Tue, 2017-06-13 01:00:25 > borg info backupBorg::2017-06-13 <-- original Name: 2017-06-13> Command line: /usr/bin/borg create -sv -C lzma,5 --exclude-from borg//backupExclude ssh://****@****///backupBorg::{now:%Y-%m-%d} /BackupSource/ Number of files: 317337 Original size Compressed size Deduplicated size This archive: 860.10 GB 664.05 GB 483.59 GB All archives: 860.10 GB 664.05 GB 483.59 GB Unique chunks Total chunks Chunk index: 464159 616383 This is to be compared to the re-created repo : > borg list backupBorgBis <-- re-created 2017-06-13 Wed, 2017-06-14 21:32:33 > borg info backupBorgBis::2017-06-13 <-- re-created Name: 2017-06-13 Command line: /usr/bin/borg create -svp -C lzma,5 backupBorgBis::2017-06-13 Backup Number of files: 317337 Original size Compressed size Deduplicated size This archive: 860.12 GB 663.59 GB 483.04 GB All archives: 860.12 GB 663.59 GB 483.04 GB Unique chunks Total chunks Chunk index: 457981 610636 The size is comparable, but not exactly the same, as is the number of chunks. Is it to be expected ? Once again, my main concern is the health of the original repo. 2) About the inability to browse fuse-mounted archive made from windows ----------------------------------------------------------------------- Upon extracting the aforementioned archive, I discovered that some of the files/folder were extracted with 077 rights (!). This seem to match the folders I was not able to browse. If needed, I'll try to reproduce or create a minimal bogus archive, and submit a bug. (I don't have access to a windows box at the time). Regards, N ----- Mail original ----- > De: nerbrume at free.fr > ?: borgbackup at python.org > Envoy?: Samedi 10 Juin 2017 13:54:04 > Objet: [Borgbackup] Windows troubles > > Hi, > > I'm having some troubles with backups made from a remote windows box. > Before submitting any bugs, I would be glad to have some feedback on > what is expected/normal. > > First, my setup : I installed borg on windows 10 using billyc's > installer > (https://github.com/borgbackup/borg/issues/440#issuecomment-279491415)(so, > cygwin), and borg is backing up on a remote linux box which also > have borg. > As the volume of data was huge, I first made two local backups from > the linux box, using a usb drive. These backups seems fine. > Later, I made several (~10) remote backup from the windows box, some > of them being partial (.checkpoint): > 2017-04-09 Sun, 2017-04-09 18:31:07 <- > local > 2017-04-12 Wed, 2017-04-12 20:02:29 <- > local > 2017-05-30.checkpoint Tue, 2017-05-30 20:39:30 <- > remote > 2017-06-01.checkpoint Thu, 2017-06-01 01:00:25 <- > remote > 2017-06-01.checkpoint.1 Thu, 2017-06-01 20:53:24 <- > remote > 2017-06-01.checkpoint.2 Thu, 2017-06-01 21:06:14 <- > remote > 2017-06-04 Sun, 2017-06-04 21:28:42 <- > remote > 2017-06-06 Tue, 2017-06-06 01:00:26 <- > remote > 2017-06-07 Wed, 2017-06-07 01:00:26 <- > remote > 2017-06-09 Fri, 2017-06-09 01:00:24 <- > remote > 2017-06-10 Sat, 2017-06-10 01:00:28 <- > remote > > The two "local" backups and the last remote one (2017-06-10) have a > original size of ~860GB > Each of the "remote" backups had a dedup. size of 500-1000 MB, and a > original size of ~940GB, due to the inclusion of a non-needed folder > (80GB containing various globally unique data, which would, I guess, > be at least 50GB after dedup+compression). > > > 1) I deleted all but the last "remote" backups (wanting to reclaim > the ~80GB lost). Before deletion: > >borg info backupBorg::2017-06-10 > Name: 2017-06-10 > Fingerprint: > d79c57fada25ab7d3e6bb07a0e642d7d28ffb704c333a56d814490c991b232d7 > Hostname: UN > Username: **** > Time (start): Sat, 2017-06-10 01:00:28 > Time (end): Sat, 2017-06-10 01:17:23 > Command line: /usr/bin/borg create -sv -C lzma,5 --exclude-from > borg//backupExclude ssh://****@****///backupBorg::{now:%Y-%m-%d} > /BackupSource/ > Number of files: 317235 > Original size Compressed size > Deduplicated size > This archive: 859.60 GB 663.66 GB > 736.28 MB > All archives: 7.73 TB 6.07 TB > 557.82 GB > Unique chunks Total chunks > Chunk index: 538363 5645695 > > And after deletion (omitting identical first lines): > >borg info backupBorg::2017-06-10 > Original size Compressed size > Deduplicated size > This archive: 859.60 GB 663.66 GB > 77.26 GB > All archives: 2.55 TB 1.99 TB > 554.58 GB > Unique chunks Total chunks > Chunk index: 532222 1851377 > > Having reclaimed only 3GB seemed suspicious, so I ran a repair : > >borg check --repair backupBorg > 132 orphaned objects found! > Archive consistency check complete, problems found. > > After that : > >borg info backupBorg::2017-06-10 > Original size Compressed size > Deduplicated size > This archive: 859.60 GB 663.66 GB > 77.26 GB > All archives: 2.55 TB 1.99 TB > 554.63 GB > Unique chunks Total chunks > Chunk index: 531596 1850784 > > How can the All archive Dedup size be bigger !? > And how come I did not recover the ~50-80GB from the non-needed > folder ? > Should I worry about the health of my repo ? > > > 2) This might be related, but mounting the "remote" backups end up in > some folders (but not all) being unreadable. > ls throws : > ls: error reading directory .: Input/output error (original in > french: "ls: lecture du r?pertoire .: Erreur d'entr?e/sortie") > while borg throws (having used borg mount -f) : > Exception after kill: > Traceback (most recent call last): > File "handlers.pxi", line 433, in llfuse.capi.fuse_releasedir > (src/llfuse/capi_linux.c:11578) > File "handlers.pxi", line 434, in llfuse.capi.fuse_releasedir > (src/llfuse/capi_linux.c:11533) > File "/usr/lib/python3/dist-packages/llfuse/pyapi.py", line 462, > in releasedir > def releasedir(self, fh): > File "/usr/lib/python3/dist-packages/borg/helpers.py", line 1290, > in handler > raise exc_cls > borg.helpers.SigTerm > Exception after kill: > Traceback (most recent call last): > File "handlers.pxi", line 406, in llfuse.capi.fuse_readdir > (src/llfuse/capi_linux.c:11058) > File "handlers.pxi", line 411, in llfuse.capi.fuse_readdir > (src/llfuse/capi_linux.c:10962) > File "misc.pxi", line 32, in llfuse.capi.fill_c_stat > (src/llfuse/capi_linux.c:16219) > OverflowError: can't convert negative value to uid_t > (last 6 lines repeated 2 times) > > However, borg check does not return any error, and so did a borg > extract --dry-run. I also hand-extracted some files (3-4), and > everything is matching the original. > Anything I might try to help devs investigate, or is windows > considered "not supported" ? > Again, should I worry about the health of my repo ? > > Thanks a bunch, > > N > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From maurice.libes at osupytheas.fr Fri Jun 23 05:49:33 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Fri, 23 Jun 2017 11:49:33 +0200 Subject: [Borgbackup] question about number of chunks Message-ID: hi to all I have an interrogation... I see differents "deduplicated size" between differents backup, I suppose that these are new chunks that had to be backuped but I see the same numbers of chunks between different backups in the stats below ... why the number of chunks don't change? or my interpretation is wrong? thanks ML borg info borg at rmyserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-22 Name: pcml-2017-06-22 Time (start): Thu, 2017-06-22 02:30:21 Time (end): Thu, 2017-06-22 02:36:20 Command line: /usr/bin/borg create -v --stats --compression zlib,3 borg at myserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-22 /home/libes /opt Number of files: 586512 Original size Compressed size Deduplicated size This archive: 295.40 GB 286.64 GB/*163.27 MB*/ All archives: 3.23 TB 3.15 TB 431.12 GB Unique chunks Total chunks /*Chunk index: 1702756 8160232*/ borg info borg at myserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-23 Name: pcml-2017-06-23 Time (start): Fri, 2017-06-23 02:30:27 Time (end): Fri, 2017-06-23 02:34:36 Command line: /usr/bin/borg create -v --stats --compression zlib,3 borg at myserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-23 /home/libes /opt Number of files: 588633 Original size Compressed size Deduplicated size This archive: 295.47 GB 286.69 GB/*225.04 MB*/ All archives: 3.23 TB 3.15 TB 431.12 GB Unique chunks Total chunks /*Chunk index: 1702756 8160232*/ -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2971 bytes Desc: Signature cryptographique S/MIME URL: From giuseppe.arvati at gmail.com Fri Jun 23 11:26:35 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Fri, 23 Jun 2017 17:26:35 +0200 Subject: [Borgbackup] borg info very long Message-ID: <912fc6b9-a986-31b8-3f9c-42b58f54a489@gmail.com> Hello , previous night a created an archive with a big file ( > 100GB ) This morning I want to check the archive and run >borg list /mnt/WSUS.borg borg response in few seconds with Using a pure-python msgpack! This will result in lower performance. WSUS-22 Thu, 2017-06-22 20:00:01 and then I tried to show info about archive WSUS-22 with borg info /mnt/WSUS.borg::WSUS-22 but after 3 hours I had no response so I interrupted the command then I tried borg check /mnt/WSUS.borg and it completed check ( 100% ) after about 50 mins and then the command continue to run without get info so after another 1hour I killed it I think isn't normal a so long delay to get results for "borg info! command and perhaps it can depend on my configuration What can I check ? Thank you From tw at waldmann-edv.de Fri Jun 23 18:31:06 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 24 Jun 2017 00:31:06 +0200 Subject: [Borgbackup] borg info very long In-Reply-To: <912fc6b9-a986-31b8-3f9c-42b58f54a489@gmail.com> References: <912fc6b9-a986-31b8-3f9c-42b58f54a489@gmail.com> Message-ID: > previous night a created an archive with a big file ( > 100GB ) You mean that a single file in your backup data source set (input files) was rather big? > This morning I want to check the archive > and run > >>borg list /mnt/WSUS.borg That just lists the manifest (all archive names/timestamps). > borg response in few seconds with > Using a pure-python msgpack! This will result in lower performance. ^ This means what it says. If you install borg from source, this often means a missing C++ compiler. The python version of msgpack is slower than the compiled C version of msgpack. > and then I tried to show info about archive WSUS-22 with > > borg info /mnt/WSUS.borg::WSUS-22 > but after 3 hours I had no response so I interrupted the command That's strange. Not sure it could be caused by just the msgpack issue or if there was something else going wrong. Are your disks ok? > then I tried > borg check /mnt/WSUS.borg > and it completed check ( 100% ) after about 50 mins > and then the command continue to run without get > info so after another 1hour I killed it borg check first does a repository (low level) check and then it checks all archives in the repository (higher level). > I think isn't normal a so long delay to get results > for "borg info! command and perhaps it can depend on > my configuration > > What can I check ? Check what's up with your msgpack. Make sure you don't get that warning by having a correctly compiled msgpack. Our docs says what additional packages you need before (re-)installing msgpack, so it gets compiled. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Fri Jun 23 18:54:59 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 24 Jun 2017 00:54:59 +0200 Subject: [Borgbackup] question about number of chunks In-Reply-To: References: Message-ID: > I see differents "deduplicated size" between differents backup, I > suppose that these are new chunks that had to be backuped Interpretation of that number for "this archive" can be difficult and the number can change over time, when you add / delete other archives. It means the summed-up size of all chunks that are ONLY used by this one archive. When other archives add references to some of these chunks, that number will decrease. When other archive remove references to some chunks also used by that archive, that number will increase. > but I see the same numbers of chunks between different backups in the > stats below ... > > why the number of chunks don't change? or my interpretation is wrong? The numbers given for the index are always overall numbers, not related to just the single archive, but to the whole chunk index (for the whole repository). > borg info borg at rmyserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-22 > > Name: pcml-2017-06-22 > > Time (start): Thu, 2017-06-22 02:30:21 > > Time (end): Thu, 2017-06-22 02:36:20 > > Command line: /usr/bin/borg create -v --stats --compression zlib,3 borg at myserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-22 /home/libes /opt > > Number of files: 586512 > > Original size Compressed size Deduplicated size > > This archive: 295.40 GB 286.64 GB /*163.27 MB*/ > > All archives: 3.23 TB 3.15 TB 431.12 GB > > Unique chunks Total chunks > > /*Chunk index: 1702756 8160232*/ > > > borg info borg at myserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-23 > > Name: pcml-2017-06-23 > > Time (start): Fri, 2017-06-23 02:30:27 > > Time (end): Fri, 2017-06-23 02:34:36 > > Command line: /usr/bin/borg create -v --stats --compression zlib,3 borg at myserver.univ.fr:/mnt/provigo-borg/sauve-pcml::pcml-2017-06-23 /home/libes /opt > > Number of files: 588633 > > Original size Compressed size Deduplicated size > > This archive: 295.47 GB 286.69 GB /*225.04 MB*/ > > All archives: 3.23 TB 3.15 TB 431.12 GB > > Unique chunks Total chunks > > /*Chunk index: 1702756 8160232*/ -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Mon Jun 26 19:03:26 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 27 Jun 2017 01:03:26 +0200 Subject: [Borgbackup] borgbackup release candidate 1.0.11rc1 released Message-ID: <0a00a653-c157-86f4-4b2d-f99c254fbd37@waldmann-edv.de> please help testing it, details see there: https://github.com/borgbackup/borg/releases/tag/1.0.11rc1 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393