From tw at waldmann-edv.de Sun Jul 2 20:00:43 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 3 Jul 2017 02:00:43 +0200 Subject: [Borgbackup] please test the beta / rc and give feedback Message-ID: We recently did some pre-releases for 1.0.11 and 1.1.0. Please help testing them and give (positive or negative) feedback via github ticket. There are the releases, the feedback github tickets are linked from the description there: https://github.com/borgbackup/borg/releases/tag/1.0.11rc1 https://github.com/borgbackup/borg/releases/tag/1.1.0b6 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tmhikaru at gmail.com Tue Jul 4 16:26:32 2017 From: tmhikaru at gmail.com (tmhikaru at gmail.com) Date: Tue, 4 Jul 2017 13:26:32 -0700 Subject: [Borgbackup] borgbackup release candidate 1.0.11rc1 released In-Reply-To: <0a00a653-c157-86f4-4b2d-f99c254fbd37@waldmann-edv.de> References: <0a00a653-c157-86f4-4b2d-f99c254fbd37@waldmann-edv.de> Message-ID: <20170704202632.GB30237@raspberrypi.home> Hmm. These fixes look very interesting, especially the hash table fixes and optimizations. I honestly can't do it right now, but I think I'll try testing borg on my raspberry pi again when I get the chance, and see if these changes make a difference in its observed misbehavior. Certainly the problems that these fixes are meant to solve are eerily similar to the stalls my rpi would get where it'd just sit there hashing at 100% cpu for days. Might be a month or two but I've definitely got to try messing with this. Tim From tw at waldmann-edv.de Wed Jul 5 09:47:05 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 5 Jul 2017 15:47:05 +0200 Subject: [Borgbackup] hashtable performance issue explained (fix in 1.0.11*) In-Reply-To: <20170704202632.GB30237@raspberrypi.home> References: <0a00a653-c157-86f4-4b2d-f99c254fbd37@waldmann-edv.de> <20170704202632.GB30237@raspberrypi.home> Message-ID: On 07/04/2017 10:26 PM, tmhikaru at gmail.com wrote: > Hmm. These fixes look very interesting, especially the hash table > fixes and optimizations. I honestly can't do it right now, but I think I'll > try testing borg on my raspberry pi again when I get the chance, and see if > these changes make a difference in its observed misbehavior. Certainly the > problems that these fixes are meant to solve are eerily similar to the > stalls my rpi would get where it'd just sit there hashing at 100% cpu for > days. Yup, likely that was the root cause of the problem you observed. I also observed it now and then, but in a less extreme way (due to faster hw). Only counting the used buckets for the fill ratio of the hash table was a mistake with major performance implications as it did not rebuild/enlarge the hashtable even if there were no empty buckets left (but everything was filled up with tombstones). That's not a problem that leads to malfunction (as in crashing or computing wrong data), but as only empty buckets terminate the search for a nonexisting key early(*), the situation escalates rather quickly and badly when the last empty buckets go away: 99 empty buckets -> has to scan ~1% of the table 9 empty buckets -> has to scan ~10% of the table 1 empty bucket -> has to scan ~50% of the table 0 empty buckets -> has to scan 100% of the table (*) And once it reached 0 empty, it was working extremely slow until it added / removed enough data to / from the hashtable to trigger a resize. When the hashtable gets resized, all data (used buckets) is used to build a new hashtable from scratch (that has no tombstones initially). Fix: now it also rebuilds the HT if there are too few empty buckets. (*) the other (unwanted, extremely slow) case for search termination is when the code notices that it has scanned the whole table. borg's hashtables are used to manage all the data/metadata chunks in the repository, so they are large - megabytes to gigabytes size. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From matej.kovacic at owca.info Sat Jul 8 18:21:12 2017 From: matej.kovacic at owca.info (Matej Kovacic) Date: Sun, 9 Jul 2017 00:21:12 +0200 Subject: [Borgbackup] a couple of questions Message-ID: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> Hi, I just discovered Borg backup and it looks really cool. Anyway, I have a couple of questions. But first, about my setup. I am running Ubuntu 16.04.2 LTS and have installed Borg 1.0.7. I am backuping to a RaspberryPi with Borg 1.0.7 installed. I installed Borg ARM build from https://borg.bauerj.eu/. I am running first (full) backup over internet and it is really slow (as expected). First I did (as normal user): borg init ssh://pi at backup.rpi/media/backup borg create --compression zlib,4 ssh://pi at backup.rpi/media/backup::matej ~ --exclude '*.vdi' --exclude ~/Downloads Since I had to set up passphrase, I understand my backup on RaspberryPi is encrpyted? Next question is: what happens if I close my laptop and put it to sleep, or if I turn it off, and then run "borg create" again? Does it continue where it has been interrupted or does it start backup all over again or something else? Next, what happens if I am using computer while it is backuping? For instance, I am adding or deleting files? What if I change a file while it is being backuped? Example of this would be, if I would be backing up VDI files (VirtualBox disks), which are large (for instance 20 GB), and while VDI file is being backuped, I am using virtual machine and changing the content of VDI file? What happens if I finish this first backup, but then run the "borg create" command again? I mean with the same repo name? Does it start the whole backup again, or will backup be incremental? Unfortunately, I am not able to see the progress. I have opened the issue, but cannot find an obvious reason for this behavior: https://github.com/borgbackup/borg/issues/2419 Are you planning to add some additional switch to show some kind of progress indicator/status? As I understand, remote borg "server" acts only as storage? I mean, I can list the status of my backups, but as I understand, system administrator cannot see the status of all machines. This could be useful for organizations, where administrator would be able to see the status of all backups and get warning for those machines which have too old backups. This could alert administrator that something is wrong with those machines. Is it possible that I backup several machines to RaspberryPi and check the status of those backups with some scripts to see which backups are OK and which failed? An my final question - are you planning to develop some GUI? I am more console guy, but I understand that for most people lack of GUI is a great obstacle. I hope my questions are not to annoying, thank you for yur answers in advance. Regards, M. -- PGP Fingerprint: 1918 8C72 E5D6 B523 86E1 AC24 C82A C043 3D92 568D PGP Key: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC82AC0433D92568D Personal blog: https://pravokator.si From adrian.klaver at aklaver.com Sat Jul 8 19:05:00 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Sat, 8 Jul 2017 16:05:00 -0700 Subject: [Borgbackup] a couple of questions In-Reply-To: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> Message-ID: <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> On 07/08/2017 03:21 PM, Matej Kovacic via Borgbackup wrote: > Hi, > > I just discovered Borg backup and it looks really cool. > > Anyway, I have a couple of questions. > > But first, about my setup. > > I am running Ubuntu 16.04.2 LTS and have installed Borg 1.0.7. I am > backuping to a RaspberryPi with Borg 1.0.7 installed. I installed Borg > ARM build from https://borg.bauerj.eu/. The current stable is 1.10. > > I am running first (full) backup over internet and it is really slow (as > expected). > > First I did (as normal user): > borg init ssh://pi at backup.rpi/media/backup > > borg create --compression zlib,4 ssh://pi at backup.rpi/media/backup::matej > ~ --exclude '*.vdi' --exclude ~/Downloads > > Since I had to set up passphrase, I understand my backup on RaspberryPi > is encrpyted? The manual is your friend. http://borgbackup.readthedocs.io/en/stable/usage.html#borg-init "If you want ?passphrase-only? security, use the repokey mode. The key will be stored inside the repository (in its ?config? file). In above mentioned attack scenario, the attacker will have the key (but not the passphrase). If you want ?passphrase and having-the-key? security, use the keyfile mode. The key will be stored in your home directory (in .config/borg/keys). In the attack scenario, the attacker who has just access to your repo won?t have the key (and also not the passphrase)." > > Next question is: what happens if I close my laptop and put it to sleep, > or if I turn it off, and then run "borg create" again? > > Does it continue where it has been interrupted or does it start backup > all over again or something else? Have never tried it. Try it a see what happens? > > Next, what happens if I am using computer while it is backuping? For > instance, I am adding or deleting files? I personally have never seen any issues running against active directories. > > What if I change a file while it is being backuped? Example of this > would be, if I would be backing up VDI files (VirtualBox disks), which > are large (for instance 20 GB), and while VDI file is being backuped, I > am using virtual machine and changing the content of VDI file? Not sure. > > What happens if I finish this first backup, but then run the "borg > create" command again? I mean with the same repo name? Does it start the > whole backup again, or will backup be incremental? Starts a new archive. Did you mean repo or archive name? Archive names need to be unique so it will error out if that is the case. If you are creating in the same repo then only new unique chunks will be added. For more detail see: http://borgbackup.readthedocs.io/en/stable/internals.html# > > Unfortunately, I am not able to see the progress. I have opened the > issue, but cannot find an obvious reason for this behavior: > https://github.com/borgbackup/borg/issues/2419 > > Are you planning to add some additional switch to show some kind of > progress indicator/status? They are already there: http://borgbackup.readthedocs.io/en/stable/usage.html# Type of log output http://borgbackup.readthedocs.io/en/stable/usage.html#borg-create -s, --stats print statistics for the created archive -p, --progress show progress display while creating the archive, showing Original, Compressed and Deduplicated sizes, followed by the Number of files seen and the path being processed, default: False --list output verbose list of items (files, dirs, ...) > > As I understand, remote borg "server" acts only as storage? I mean, I > can list the status of my backups, but as I understand, system > administrator cannot see the status of all machines. This could be > useful for organizations, where administrator would be able to see the > status of all backups and get warning for those machines which have too > old backups. This could alert administrator that something is wrong with > those machines. Is it possible that I backup several machines to > RaspberryPi and check the status of those backups with some scripts to > see which backups are OK and which failed? http://borgbackup.readthedocs.io/en/stable/usage.html#borg-check http://borgbackup.readthedocs.io/en/stable/usage.html#borg-list http://borgbackup.readthedocs.io/en/stable/usage.html#borg-info I do my backups via cron so I get an email that shows success or failure. > > An my final question - are you planning to develop some GUI? I am more > console guy, but I understand that for most people lack of GUI is a > great obstacle. A simple GUI that needs more work: https://github.com/borgbackup/borgweb > > I hope my questions are not to annoying, thank you for yur answers in > advance. Some time with the manual would have made the post shorter:) > > Regards, > M. > -- Adrian Klaver adrian.klaver at aklaver.com From draget at speciesm.net Sun Jul 9 04:10:45 2017 From: draget at speciesm.net (Draget) Date: Sun, 9 Jul 2017 10:10:45 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> Message-ID: Hey there, :-) > The current stable is 1.10. Just to avoid confusion: 1.0.10 (and very soon 1.0.11) Your version 1.0.7 should do it's job just fine. But there have been a lot of cosmetic / performance and mintor corner-case bugfixes. There is no danger in just replacing it with 1.0.10, the backups will work the same. > If you want ?passphrase and having-the-key? security, use the keyfile > mode. The key will be stored in your home directory (in > .config/borg/keys). In the attack scenario, the attacker who has just > access to your repo won?t have the key (and also not the passphrase)." Small note: In keyfile mode you are responsible for keeping the keyfile. Do not keep it on the PC you back up (since then it is gone when the PC crashes as you cannot access you backup). In most cases a repokey with a strong passphrase is the more sane choice. >> Next question is: what happens if I close my laptop and put it to sleep, >> or if I turn it off, and then run "borg create" again? >> >> Does it continue where it has been interrupted or does it start backup >> all over again or something else? > Have never tried it. Try it a see what happens? Borg is explicitly desinged to take care of that! It creates a 'checkpoit' of it's index every 5 minutes (changable via --checkpoint-interval). If you connection drops (or long hibernate) or PC crashes while backing up, all chunks up to the last checkpoint will be in the index. You snapshot you tried to created in the repository will only be saved as '.checkpoint' and if you run the command again, borg will skip all already known chunks, backup the rest and if successfull, remove the old .checkpoint snapshot for you (if the name was the same). So, you can happily interrupt your backup (if it ran for at least 5-10 minutes) and run it again until you managed to save everything. Perfect for slow, initial backups over the internet. (Note: Ofcourse, you should not nessecarily stress-test this feature by forcing it a thousand times on your production-critical data? xD) >> Next, what happens if I am using computer while it is backuping? For >> instance, I am adding or deleting files? Borg usues regular filesystem syscalls to walk though your files. If it has not worked through directory X yet and you change something, borg will backup this change once it is in directory X. The other way around, borg will not notice any changes done to directories it already processed. When working on a file WHILE borg is chunking it? it might cause locking issues (borg will warn and skip it) or in the worst-case (if not locked), it will store an inconsistent state. It is your responsibility to avoid that. Generally speaking, for a regular desktop-user this is a very rare case. Most of the time programms lock their files or save them in a burst. And even if, the next backup the consistent version will be saved. So for regular desktop usage you will not have any issue in 99% of the cases if you work while backup up - tho simply fix that by doing your backup while you make a pause or at night. :-P >> What if I change a file while it is being backuped? Example of this >> would be, if I would be backing up VDI files (VirtualBox disks), which >> are large (for instance 20 GB), and while VDI file is being backuped, I >> am using virtual machine and changing the content of VDI file? For stuff like (active) virtual machine images (borg loves those, deduplication is *great* on these) ofcourse the chance is a little higher as they are constantly written to. Same with database files and such. For critical data, you should make sure to have a consistant sate. Dumps (DB-Dumps, VM-Snapshots) or on filesystem-level (btfs or zfs snapshot) or on block level (lvm snapshot) for borg to work on. (That being said, I have an active VM on my homeserver which I backup 'the wrong way' without snapshots or pausing it. Did one test and had one actual restore - both times it worked without issues. Tho the data is not exactly super-critical, that's why I do it. Sitll not recommended. :P) >> Unfortunately, I am not able to see the progress. I have opened the >> issue, but cannot find an obvious reason for this behavior: >> https://github.com/borgbackup/borg/issues/2419 >> >> Are you planning to add some additional switch to show some kind of >> progress indicator/status? --verbose --progress? but I think this is one of the cosmetic things that has been improved in newer versions. :-P Happying backupping! :-) Michael From adrian.klaver at aklaver.com Sun Jul 9 10:28:14 2017 From: adrian.klaver at aklaver.com (Adrian Klaver) Date: Sun, 9 Jul 2017 07:28:14 -0700 Subject: [Borgbackup] a couple of questions In-Reply-To: References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> Message-ID: On 07/09/2017 01:10 AM, Draget wrote: > Hey there, :-) > >> The current stable is 1.10. > > Just to avoid confusion: 1.0.10 (and very soon 1.0.11) > My mistake, thanks for catching that. -- Adrian Klaver adrian.klaver at aklaver.com From maurice.libes at osupytheas.fr Wed Jul 12 06:22:10 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Wed, 12 Jul 2017 12:22:10 +0200 Subject: [Borgbackup] encryption mechanism understanding Message-ID: <19e3dbe7-6a9b-dc1d-89e2-a16451c7f852@osupytheas.fr> hi to all for my understanding may I have some basics elements about the encryption process between the server and the client ? in the "repokey" mode, the encryption key is stored in the config file of one repository, ok? but the chunk encryption is done on the client, right? so how the client is able to encrypt the chunks with the key which is stored on the server? does this mean that the key is sent on the network towards the client ? On the client side how is used the passphrase? It serves to access the encryption key on the server? does this means that the passphrase is sent to the borg server to access the encryption key of one repository? Does the client needs the encryption key stored on the server? In short I don't understand the different accesses made by the client and server to the key and the passphrase If one client PC crashes or burns, can I restore my data from another PC with only the passphrase sent to the borg server ? If my Borg server crashes (without the NAS containing the backup repositories) , may I access the backuped data again, with only a backup of the differents encryption keys of the repositories (borg key export) sorry if it is too long to explain many thanks for any answers ML ps: We make a communication about borgbackup in a french congress in November https://www.jres.org/fr/programme and I need to clarify some issues -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From tw at waldmann-edv.de Wed Jul 12 12:24:39 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 12 Jul 2017 18:24:39 +0200 Subject: [Borgbackup] encryption mechanism understanding In-Reply-To: <19e3dbe7-6a9b-dc1d-89e2-a16451c7f852@osupytheas.fr> References: <19e3dbe7-6a9b-dc1d-89e2-a16451c7f852@osupytheas.fr> Message-ID: >in the "repokey" mode, the encryption key is stored in the config file of the repository, ok? the passphrase-protected/encrypted key, yes. >but the chunk encryption is done on the client yes. >so how the client is able to encrypt the chunks with the key which is >stored on the server? >does this mean that the key is sent on the network towards the client ? the passphrase-protected/encrypted key, yes. >On the client side how is used the passphrase? it is used to decrypt the key. >does this means that the passphrase is sent to the borg server to >access >the encryption key of one repository? no. >Does the client needs the encryption key stored on the server? Sure. >If one client PC crashes or burns, can I restore my data from another >PC with only the passphrase? yes. Assuming that the repokey is still in the repo config. >If my Borg server crashes (without the NAS containing the backup >repositories) , may I access the backuped data again, with only a >backup >of the differents encryption keys of the repositories (borg key >export) if you use repokey, keys are in repo config. you also need the passphrase. you should always have a backup of keys, but here you won't need it. >ps: We make a communication about borgbackup in a french congress in >November >https://www.jres.org/fr/programme >and I need to clarify some issues Great. If you have more questions or you'ld like review of slides, just ask. -- Sent from my Android device with K-9 Mail. Please excuse my brevity. From public at enkore.de Wed Jul 12 12:43:29 2017 From: public at enkore.de (Marian Beermann) Date: Wed, 12 Jul 2017 18:43:29 +0200 Subject: [Borgbackup] encryption mechanism understanding In-Reply-To: <19e3dbe7-6a9b-dc1d-89e2-a16451c7f852@osupytheas.fr> References: <19e3dbe7-6a9b-dc1d-89e2-a16451c7f852@osupytheas.fr> Message-ID: <7a346f94-6beb-729f-6586-89b674642b8e@enkore.de> Docs: https://borgbackup.readthedocs.io/en/latest/internals/security.html#cryptography-in-borg https://borgbackup.readthedocs.io/en/latest/internals/data-structures.html#encryption From tr.ml at gmx.de Thu Jul 13 04:48:54 2017 From: tr.ml at gmx.de (Rainer Traut) Date: Thu, 13 Jul 2017 10:48:54 +0200 Subject: [Borgbackup] borgbackup release candidate 1.0.11rc1 released In-Reply-To: <0a00a653-c157-86f4-4b2d-f99c254fbd37@waldmann-edv.de> References: <0a00a653-c157-86f4-4b2d-f99c254fbd37@waldmann-edv.de> Message-ID: <027d6ecf-d1f5-72b9-13b3-e0c5f729fd7c@gmx.de> Am 27.06.2017 um 01:03 schrieb Thomas Waldmann: > please help testing it, details see there: > > https://github.com/borgbackup/borg/releases/tag/1.0.11rc1 > I've been using this version to create a repository and to backup some ssh mounted hosts. Works w/o problem on RHEL6 by using compression lz4. VG Rainer From contact at devetdadam.com Thu Jul 13 06:11:44 2017 From: contact at devetdadam.com (Contact DEV & d'Adam) Date: Thu, 13 Jul 2017 12:11:44 +0200 Subject: [Borgbackup] Record logs Message-ID: Dear list, Is there a simple way to invoque in the commande line a log file where logs would be detailed. I've looking for this in the doc, but couldn't find the right trick. Best regards ! JMarc From tw at waldmann-edv.de Thu Jul 13 06:52:36 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 13 Jul 2017 12:52:36 +0200 Subject: [Borgbackup] Record logs In-Reply-To: References: Message-ID: > Is there a simple way to invoque in the commande line a log file where > logs would be detailed. > > I've looking for this in the doc, but couldn't find the right trick. -v ? -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From contact at devetdadam.com Thu Jul 13 06:58:20 2017 From: contact at devetdadam.com (Contact DEV & d'Adam) Date: Thu, 13 Jul 2017 12:58:20 +0200 Subject: [Borgbackup] Record logs In-Reply-To: References: Message-ID: Sorry : borg 1.0.10 Le 13/07/2017 ? 12:52, Thomas Waldmann a ?crit : >> Is there a simple way to invoque in the commande line a log file where >> logs would be detailed. >> >> I've looking for this in the doc, but couldn't find the right trick. > -v ? > > From tw at waldmann-edv.de Thu Jul 13 07:03:43 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 13 Jul 2017 13:03:43 +0200 Subject: [Borgbackup] Record logs In-Reply-To: References: Message-ID: >>> Is there a simple way to invoque in the commande line a log file where >>> logs would be detailed. >>> >>> I've looking for this in the doc, but couldn't find the right trick. >> -v ? -v is the option for verbose. querying the version is --version, btw. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From contact at devetdadam.com Thu Jul 13 08:22:15 2017 From: contact at devetdadam.com (Contact DEV & d'Adam) Date: Thu, 13 Jul 2017 14:22:15 +0200 Subject: [Borgbackup] Record logs In-Reply-To: References: Message-ID: :-D ok - some improvement on the way... Well, -v is ok ,but i'd like to have a .log file with all the messages in it for a backup report. Any idea ? Le 13/07/2017 ? 13:03, Thomas Waldmann a ?crit : >>>> Is there a simple way to invoque in the commande line a log file where >>>> logs would be detailed. >>>> >>>> I've looking for this in the doc, but couldn't find the right trick. >>> -v ? > -v is the option for verbose. querying the version is --version, btw. > > > From maurice.libes at osupytheas.fr Thu Jul 13 08:39:30 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Thu, 13 Jul 2017 14:39:30 +0200 Subject: [Borgbackup] Record logs In-Reply-To: References: Message-ID: Le 13/07/2017 ? 14:22, Contact DEV & d'Adam a ?crit : > :-D ok - some improvement on the way... > Well, -v is ok ,but i'd like to have a .log file with all the messages > in it for a backup report. > Any idea ? may be I don't understand what you want , but to gather the logs into a logfile, you just have to redirect the standard output of each borg command no? like that $BORG create \ -v --stats \ $COMPRESSION \ $BORG_ARCHIVE \ $REP_ASAUVEGARDER \ --exclude '/home/*/.cache' \ >> ~/BORG/backup.log 2>&1 $BORG prune -v --list --info --keep-daily=7 --keep-weekly=2 \ $BORG_REPOSITORY --prefix ${hostname} \ >>~/BORG/backup.log 2>&1 etc. > Le 13/07/2017 ? 13:03, Thomas Waldmann a ?crit : >>>>> Is there a simple way to invoque in the commande line a log file >>>>> where >>>>> logs would be detailed. >>>>> >>>>> I've looking for this in the doc, but couldn't find the right trick. >>>> -v ? >> -v is the option for verbose. querying the version is --version, btw. >> >> >> > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From contact at devetdadam.com Thu Jul 13 09:07:31 2017 From: contact at devetdadam.com (Contact DEV & d'Adam) Date: Thu, 13 Jul 2017 15:07:31 +0200 Subject: [Borgbackup] Record logs In-Reply-To: References: Message-ID: <070373dd-74c2-b51a-e502-8ab58ddc4cba@devetdadam.com> Many thanks : that was the trick ! Best regards, JM Le 13/07/2017 ? 14:39, Maurice Libes a ?crit : > > > Le 13/07/2017 ? 14:22, Contact DEV & d'Adam a ?crit : >> :-D ok - some improvement on the way... >> Well, -v is ok ,but i'd like to have a .log file with all the >> messages in it for a backup report. >> Any idea ? > > may be I don't understand what you want , but > to gather the logs into a logfile, you just have to redirect the > standard output of each borg command > no? > > like that > > $BORG create \ > -v --stats \ > $COMPRESSION \ > $BORG_ARCHIVE \ > $REP_ASAUVEGARDER \ > --exclude '/home/*/.cache' \ > >> ~/BORG/backup.log 2>&1 > > $BORG prune -v --list --info --keep-daily=7 --keep-weekly=2 \ > $BORG_REPOSITORY --prefix ${hostname} \ > >>~/BORG/backup.log 2>&1 > > etc. > >> Le 13/07/2017 ? 13:03, Thomas Waldmann a ?crit : >>>>>> Is there a simple way to invoque in the commande line a log file >>>>>> where >>>>>> logs would be detailed. >>>>>> >>>>>> I've looking for this in the doc, but couldn't find the right trick. >>>>> -v ? >>> -v is the option for verbose. querying the version is --version, btw. >>> >>> >>> >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- An HTML attachment was scrubbed... URL: From _ at thomaslevine.com Fri Jul 14 06:05:27 2017 From: _ at thomaslevine.com (Thomas Levine) Date: Fri, 14 Jul 2017 10:05:27 +0000 Subject: [Borgbackup] Some prototype features Message-ID: <1500026727.1150248.1040701368.0D126443@webmail.messagingengine.com> Borg lacks many features in fossil that I really like, so I have written rough equivalents of my favorites for my own use. Has someone already written them better? The important features are * Configuration of repository access settings with ``torg open`` and ``.trgckout`` * Using settings from a checkout-specific file instead of from the shell arguments * Run commands on all repositories with ``torg all``, like in ``fossil all`` If not, maybe they can at least inspire something. If repository settings are stored more formally, we can have things like a checkout-specific default compression and global default excludes file. Then creating an archive could be even shorter than it already is, like just "borg create name-of-archive" (inspired by "fossil commit -m commit-message"). And if checkouts and repositories are indexed centrally, we can easily run commands on all of them at once. I use fossil for versioning of small things and borg for versioning of everything else. With the two features above, my backup procedure gets much easier. First I append new commits in the fossil repositories to their remotes. $ fossil all push Then I create new borg archives and append them to their remotes. $ torg all create ARCHIVE-NAME My implementation is here. https://thomaslevine.com/scm/torg/artifact/8580af67aee9b738 From ghs.cnt at gmail.com Tue Jul 18 09:12:11 2017 From: ghs.cnt at gmail.com (Gigi Canuto) Date: Tue, 18 Jul 2017 15:12:11 +0200 Subject: [Borgbackup] Deduplication and encryption Message-ID: Hello, I'm gigi, IT student. I am interested in using Borg as a backup system, so I am documenting myself about the features it has. I am concerned about security and privacy, encryption is a must. Also because probably I'll need to backup a lot of machines I need deduplication and (eventually) compression. I wondered how Borg could handle deduplication and encryption together, so I asked this question on superuser.com forum. Could you please confirm that the answer is correct ? Can you provide a more detailed explaination of how Borg handles this kind of situation ? I know that the documentation already states the encryption is done client side, but could you confirm it ? I will for sure test the cited features, but your explainations could help me a lot. Thanks! Have a nice day, gigi canuto -------------- next part -------------- An HTML attachment was scrubbed... URL: From matej.kovacic at owca.info Wed Jul 19 06:39:46 2017 From: matej.kovacic at owca.info (Matej Kovacic) Date: Wed, 19 Jul 2017 12:39:46 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> Message-ID: Hi, > Your version 1.0.7 should do it's job just fine. But there have been a > lot of cosmetic / performance and mintor corner-case bugfixes. There is > no danger in just replacing it with 1.0.10, the backups will work the same. I upgraded to 1.0.10 (on my Ubuntu client AND RaspberryPi server machines), and run: borg upgrade --tam ssh://pi at backup.rpi/media/backup/matej Now I am running a check. This should be OK? BTW, I was running my first backup for a week, with several interrupts. Now when I say borg list ssh://pi at backup.rpi/media/backup/matej I got this: matej_prenosnik.checkpoint Sat, 2017-07-08 16:22:48 matej_prenosnik.checkpoint.1 Sat, 2017-07-08 18:36:45 matej_prenosnik.checkpoint.2 Sat, 2017-07-08 20:30:03 matej_prenosnik.checkpoint.3 Sun, 2017-07-09 12:05:01 matej_prenosnik.checkpoint.4 Sun, 2017-07-09 14:38:42 matej_prenosnik.checkpoint.5 Mon, 2017-07-10 08:42:53 matej_prenosnik.checkpoint.6 Mon, 2017-07-10 15:26:04 matej_prenosnik.checkpoint.7 Tue, 2017-07-11 16:11:21 matej_prenosnik.checkpoint.8 Wed, 2017-07-12 10:05:50 matej_prenosnik.checkpoint.9 Thu, 2017-07-13 08:10:57 matej_prenosnik.checkpoint.10 Thu, 2017-07-13 11:23:32 matej_prenosnik.checkpoint.11 Fri, 2017-07-14 09:08:25 matej_prenosnik.checkpoint.12 Fri, 2017-07-14 17:13:20 matej_prenosnik.checkpoint.13 Fri, 2017-07-14 22:24:08 matej_start Sat, 2017-07-15 10:54:40 matej_17-07-2017.checkpoint Mon, 2017-07-17 15:44:55 matej_17-07-2017 Mon, 2017-07-17 16:50:06 At some point I renamed matej_presnosnik to matej_start. Anyway, I can see several checkpoints. Is it possible to "merge" them on a server side? And a final question, I would like to run Borg automatically for instance, every day. I assume, I can add to cron someting like this: borg create /path/to/repo::{hostname}-{user}-{now:%Y-%m-%dT%H:%M:%S} However, what happens if I lose my internet connection? And then next day, backup will be restarted with new name? Regards, M. -- PGP Fingerprint: 1918 8C72 E5D6 B523 86E1 AC24 C82A C043 3D92 568D PGP Key: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC82AC0433D92568D Personal blog: https://pravokator.si From tw at waldmann-edv.de Wed Jul 19 08:00:47 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 19 Jul 2017 14:00:47 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> Message-ID: > matej_17-07-2017.checkpoint Mon, 2017-07-17 15:44:55 > matej_17-07-2017 Mon, 2017-07-17 16:50:06 > > At some point I renamed matej_presnosnik to matej_start. Anyway, I can > see several checkpoints. Is it possible to "merge" them on a server side? Usually one deletes the checkpoints once one has a completed backup of same stuff. Borg 1.1 will have code in borg prune that automatically deals with deleting all checkpoints (except the latest one, when it is not superceded by a non-checkpoint backup). > And a final question, I would like to run Borg automatically for > instance, every day. I assume, I can add to cron someting like this: > > borg create /path/to/repo::{hostname}-{user}-{now:%Y-%m-%dT%H:%M:%S} Yes, add that to cron. And maybe add some excludes for stuff you don't want to backup. And also a call to borg prune like: borg prune --prefix {hostname}-{user}- ... > However, what happens if I lose my internet connection? And then next > day, backup will be restarted with new name? Yes, borg always does full backups. It never does incremental backups and it also never "resumes" from something that was interrupted - it just *feels* like that because it is fast due to deduplication. The name does not matter for retrying an interrupted backup except that it must not be an existing name, so including the timestamp in the name solves that. It makes a checkpoint (see --checkpoint-interval) now and then, which is a valid, but incomplete backup. If borg gets interrupted, everything after the last checkpoint will be a incomplete transaction and get rolled back next time you use the repo. The checkpoint archive contains references to all the chunks it processed until the checkpoint time, so it won't need to transfer these again (it notices it already has them in the repo and just refers to them in the next backup) - that's why it feels so fast. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From matej.kovacic at owca.info Wed Jul 19 16:10:24 2017 From: matej.kovacic at owca.info (Matej Kovacic) Date: Wed, 19 Jul 2017 22:10:24 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> Message-ID: <145b12dd-8d9e-bd13-f623-cc674961a8f5@owca.info> Hi, > Usually one deletes the checkpoints once one has a completed backup of > same stuff. Probably stupid queston, but how to do that? borg delete is probably not the correct option? > Borg 1.1 will have code in borg prune that automatically deals with > deleting all checkpoints (except the latest one, when it is not > superceded by a non-checkpoint backup). Nice to hear that. Any plans when it is going to be released? > And also a call to borg prune like: > > borg prune --prefix {hostname}-{user}- ... OK, as I understand, borg prune with -d 7 will keep only backups for last 7 days. OK, for instance, I put into cron a script with three commands: - set a variable with current time - rum borg create with repo name with this time - run borg prune with --prefix and this time What if I loose connection while running borg create, but borg prune is run? And, what happens, if I do not run borg create for a month, and then run borg prune -d 7? Are all data lost, since all backups are older than 7 days? > Yes, borg always does full backups. It never does incremental backups > and it also never "resumes" from something that was interrupted - it Ah, so when mounting a repo, I always get the last backuped version? So I cannot see several versions of a same file for instance for each day of a week? Regards, M. -- PGP Fingerprint: 1918 8C72 E5D6 B523 86E1 AC24 C82A C043 3D92 568D PGP Key: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC82AC0433D92568D Personal blog: https://pravokator.si From tw at waldmann-edv.de Wed Jul 19 17:21:46 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 19 Jul 2017 23:21:46 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: <145b12dd-8d9e-bd13-f623-cc674961a8f5@owca.info> References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> <145b12dd-8d9e-bd13-f623-cc674961a8f5@owca.info> Message-ID: >> Usually one deletes the checkpoints once one has a completed backup of >> same stuff. > Probably stupid queston, but how to do that? > > borg delete is probably not the correct option? it is the correct option. checkpoint archives are normal, valid, but incomplete archives. in 1.1, you will have the additional option to just wait, the next prune will kill most or all checkpoints. >> Borg 1.1 will have code in borg prune that automatically deals with >> deleting all checkpoints (except the latest one, when it is not >> superceded by a non-checkpoint backup). > Nice to hear that. Any plans when it is going to be released? there soon will be a release candidate. currently you can use beta 6 for careful testing. >> borg prune --prefix {hostname}-{user}- ... > OK, as I understand, borg prune with -d 7 will keep only backups for > last 7 days. prune is not super easy to understand (nor to explain) in a precise way, so please see the prune docs and also the example in docs/misc/... to understand it. > > OK, for instance, I put into cron a script with three commands: > - set a variable with current time > - rum borg create with repo name with this time you can use {now} and you won't need the variable with the current time. > - run borg prune with --prefix and this time you don't need to give the time to prune at all, it will use the system clock. > What if I loose connection while running borg create, but borg prune is run? You will have a checkpoint from the interrupted create run. In 1.0, borg prune will completely ignore all checkpoints and just work on the non-checkpoint archives. In 1.1 it will keep a checkpoint archive if it is not already superceded by an non-checkpoint archive, but delete all other (older) checkpoints. > And, what happens, if I do not run borg create for a month, and then run > borg prune -d 7? Are all data lost, since all backups are older than 7 days? Read the docs, please. >> Yes, borg always does full backups. It never does incremental backups >> and it also never "resumes" from something that was interrupted - it > Ah, so when mounting a repo, I always get the last backuped version? So > I cannot see several versions of a same file for instance for each day > of a week? borg mount with a repo will show you one directory per archive that is named like the archive and has the content of that archive. -- GPG Fingerprint: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt. From heiko.helmle at horiba.com Thu Jul 20 10:30:28 2017 From: heiko.helmle at horiba.com (heiko.helmle at horiba.com) Date: Thu, 20 Jul 2017 16:30:28 +0200 Subject: [Borgbackup] Official Windows Support Message-ID: Hello, it looks to me that the windows branch has become pretty quiet. I'm sorry to see that - i'd be quite interested in a command line borg that would work natively (without cygwin) on windows. So a question from a bystander: How far away would (basic) windows support be? I'm not expecting full features (ACLs or VSS-Support). Just being able to backup files to a central location with (preferably) native path syntax and as few workarounds as possible. Is something like that (borg running on native windows python) even remotely planned? Best Regards Heiko -------------- next part -------------- An HTML attachment was scrubbed... URL: From matej.kovacic at owca.info Thu Jul 20 10:47:11 2017 From: matej.kovacic at owca.info (Matej Kovacic) Date: Thu, 20 Jul 2017 16:47:11 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> <145b12dd-8d9e-bd13-f623-cc674961a8f5@owca.info> Message-ID: Hi, >> borg delete is probably not the correct option? > > it is the correct option. So: borg delete ssh://pi at backup.rpi/media/backup/matej::matej_laptop.checkpoint.1 -p -s and then ::matej_laptop..checkpoint.2, etc. till checkpoint.13 (which is the last)? Is it possible to "clear" all checckpoints at once? Regards, M. -- PGP Fingerprint: 1918 8C72 E5D6 B523 86E1 AC24 C82A C043 3D92 568D PGP Key: https://keyserver.ubuntu.com/pks/lookup?op=get&search=0xC82AC0433D92568D Personal blog: https://pravokator.si From public at enkore.de Thu Jul 20 10:49:07 2017 From: public at enkore.de (Marian Beermann) Date: Thu, 20 Jul 2017 16:49:07 +0200 Subject: [Borgbackup] Official Windows Support In-Reply-To: References: Message-ID: <2f1e72c0-6bfa-529d-d29f-7449a3601938@enkore.de> On 20.07.2017 16:30, heiko.helmle at horiba.com wrote: > Hello, > > it looks to me that the windows branch has become pretty quiet. I'm > sorry to see that - i'd be quite interested in a command line borg that > would work natively (without cygwin) on windows. > > So a question from a bystander: How far away would (basic) windows > support be? > > I'm not expecting full features (ACLs or VSS-Support). Just being able > to backup files to a central location with (preferably) native path > syntax and as few workarounds as possible. Is something like that (borg > running on native windows python) even remotely planned? > > Best Regards > Heiko > What you describe pretty much worked last time I checked, but it sometimes crashed on some folders/directories. m From tw at waldmann-edv.de Thu Jul 20 14:20:00 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 20 Jul 2017 20:20:00 +0200 Subject: [Borgbackup] a couple of questions In-Reply-To: References: <7f2815a0-1b88-8f65-cfe2-ec5446ac20ff@owca.info> <2d973d7d-c27f-bfca-6abd-7d4a83a2a94d@aklaver.com> <145b12dd-8d9e-bd13-f623-cc674961a8f5@owca.info> Message-ID: > Is it possible to "clear" all checckpoints at once? As I said, borg 1.1 will do that in prune (but not 1.0.x). In case you need to automate, a little script with borg list --short, grep and borg delete should do the trick. Be careful. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Thu Jul 20 14:32:03 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 20 Jul 2017 20:32:03 +0200 Subject: [Borgbackup] Official Windows Support In-Reply-To: References: Message-ID: > it looks to me that the windows branch has become pretty quiet. Yes, it needs more developers capable of (and willing to) writing and reviewing windows-specific code. > sorry to see that - i'd be quite interested in a command line borg that > would work natively (without cygwin) on windows. Sure, that would be nice. It just needs to be done, to be reviewed, to be tested. Before it can get merged, there should be somehow a long term maintenance perspective, like more than 1 developer working on the code every few months. Also, borg's core code should not be uglified by windows-specific additions, they should be kept mostly separate in platform-specific modules. In the issue tracker, there's also some work todo, like splitting big "overall" tickets into smaller well-specified ones, so bounties can be put on them. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Thu Jul 20 14:41:30 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 20 Jul 2017 20:41:30 +0200 Subject: [Borgbackup] Official Windows Support In-Reply-To: References: Message-ID: <02ee1f86-7628-da2d-9ded-249f0ffe2c4d@waldmann-edv.de> Another issue is testing. For the UNIX-like platforms, we usually have some CI (travis-ci for Linux and OS X) that tests every PR before merging. The windows branch has AppVeyor for CI, 53 test failures currently. There are also vagrant VMs that can be used for local / more practical testing. For windows (currently: only with cygwin), there's a vagrant machine, but it lacks quite some usability / automation. So, quite some TODO... -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Fri Jul 21 11:58:58 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 21 Jul 2017 17:58:58 +0200 Subject: [Borgbackup] borgbackup 1.0.11 released Message-ID: Bugfix release, details see there: https://github.com/borgbackup/borg/releases/tag/1.0.11 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From sebastian.felis at gmx.de Fri Jul 21 12:59:21 2017 From: sebastian.felis at gmx.de (Sebastian Felis) Date: Fri, 21 Jul 2017 18:59:21 +0200 Subject: [Borgbackup] borg recreate with changed segments_per_dir? Message-ID: <377f180a-a64d-f699-7323-a65637e4b5d3@gmx.de> Hi, first thank you for the awesome backup solution borg! It is the best backup software I ever had. I have a question regarding segments_per_dir. I am running borg 1.0.9 from Debian Jessie backports with default segments_per_dir of 10000. Now I would like to store my ciphered backup online but my storage provider allows only 3000 file per directory. Is there a way to recreate the repository data structure with changed settings of segments_per_dir? Or maybe there is a low level api to rewrite single chunks manually? Thank you in advance Sebastian From tw at waldmann-edv.de Fri Jul 21 16:24:45 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 21 Jul 2017 22:24:45 +0200 Subject: [Borgbackup] borg recreate with changed segments_per_dir? In-Reply-To: <377f180a-a64d-f699-7323-a65637e4b5d3@gmx.de> References: <377f180a-a64d-f699-7323-a65637e4b5d3@gmx.de> Message-ID: <9ea391a6-5c58-5efa-b090-59d03b8726ca@waldmann-edv.de> > Is there a way to recreate the repository data structure with changed > settings of segments_per_dir? Not yet. This has just come up on githb, see there: https://github.com/borgbackup/borg/pull/2846 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From sebastian.felis at gmx.de Sat Jul 22 03:18:48 2017 From: sebastian.felis at gmx.de (Sebastian Felis) Date: Sat, 22 Jul 2017 09:18:48 +0200 Subject: [Borgbackup] borg recreate with changed segments_per_dir? In-Reply-To: <9ea391a6-5c58-5efa-b090-59d03b8726ca@waldmann-edv.de> References: <377f180a-a64d-f699-7323-a65637e4b5d3@gmx.de> <9ea391a6-5c58-5efa-b090-59d03b8726ca@waldmann-edv.de> Message-ID: On 21.07.2017 22:24, Thomas Waldmann wrote: >> Is there a way to recreate the repository data structure with changed >> settings of segments_per_dir? > > Not yet. > > This has just come up on githb, see there: > > https://github.com/borgbackup/borg/pull/2846 Thank you for the quick reply and the link. It seems that a automated relocation of segments wont be supported. But the directory structure of segments seems to be simple. Please correct me if I am wrong: tl;dr: data// where segment_dir is floor( / ) Borg writes segments which ids (or names) are simple counter with offset of 1 for the next segment. The directory is created by dividing the current segment id by the segment_per_dir value. So if I change the segment_per_dir value, I have to iterate through all segment ids, calculate the new segment_dir and move each segment file to the new calculated directory. Example: old segment_per_dir is 10000, new segment_per_dir is 450; for segment 475332 the original segment dir would be 47 while the new one would be 1056 (475332 / 450). If so, it would be fair simple to change segments_per_dir manually. Sebastian From sebastian.felis at gmx.de Sun Jul 23 07:49:02 2017 From: sebastian.felis at gmx.de (Sebastian Felis) Date: Sun, 23 Jul 2017 13:49:02 +0200 Subject: [Borgbackup] borg recreate with changed segments_per_dir? In-Reply-To: References: <377f180a-a64d-f699-7323-a65637e4b5d3@gmx.de> <9ea391a6-5c58-5efa-b090-59d03b8726ca@waldmann-edv.de> Message-ID: <20c0d90b-1fba-a154-a8cf-1a24a6992cb4@gmx.de> On 22.07.2017 09:18, Sebastian Felis wrote: > On 21.07.2017 22:24, Thomas Waldmann wrote: >>> Is there a way to recreate the repository data structure with changed >>> settings of segments_per_dir? >> >> Not yet. >> >> This has just come up on githb, see there: >> >> https://github.com/borgbackup/borg/pull/2846 > > Thank you for the quick reply and the link. It seems that a automated > relocation of segments wont be supported. > > > But the directory structure of segments seems to be simple. Please > correct me if I am wrong: > > tl;dr: > > data// > > where segment_dir is floor( / ) > > > Borg writes segments which ids (or names) are simple counter with offset > of 1 for the next segment. The directory is created by dividing the > current segment id by the segment_per_dir value. > > So if I change the segment_per_dir value, I have to iterate through all > segment ids, calculate the new segment_dir and move each segment file to > the new calculated directory. > > > Example: > > old segment_per_dir is 10000, new segment_per_dir is 450; > > for segment 475332 the original segment dir would be 47 while the new > one would be 1056 (475332 / 450). > > > If so, it would be fair simple to change segments_per_dir manually. I wrote a small script to relocate the segments if segments_per_dir was changed: https://gist.github.com/xemle/4ac5c5d65d22b828198c32b0ac58a07c borg check reported no issues after relocation Cheers, Sebastian From tw at waldmann-edv.de Sun Jul 23 19:45:45 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 24 Jul 2017 01:45:45 +0200 Subject: [Borgbackup] borgbackup 1.1.0 release candidate 1 released Message-ID: <1ee7417c-9989-fba3-810a-90ec4bbd1f37@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.1.0rc1 please help testing and provide feedback. details see url. cheers, thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From mfseeker at gmail.com Fri Jul 28 06:18:33 2017 From: mfseeker at gmail.com (Stan Armstrong) Date: Fri, 28 Jul 2017 07:18:33 -0300 Subject: [Borgbackup] Bare Metal Backup? In-Reply-To: References: Message-ID: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> Is there a straightforward way of using borgbackup to do a full metal backup/restore of my system partition? I am running Debian Stretch 9.1. I would like to back it up in such a way that if my SSD boot drive is destroyed, I can restore my system immediately in bootable condition. All of the solutions I have located are too complex for my aging brain. I have understood borg well enough to use it for many months, backing up various partitions to and from other drives both locally and via SSH on my household LAN. I automated everything by writing simple scripts and using cron. I suspect that I simply don't know how to make a restored partition bootable. From gait at ATComputing.nl Fri Jul 28 06:36:44 2017 From: gait at ATComputing.nl (Gerrit A. Smit) Date: Fri, 28 Jul 2017 12:36:44 +0200 Subject: [Borgbackup] Bare Metal Backup? In-Reply-To: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> References: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> Message-ID: <57841c9f-c4e4-d55c-946e-13ae443aa9ef@ATComputing.nl> Hallo, ReaR has integrated support for Borg Although it does not show op as a feature, I use it (not fully tested by me yet): http://relax-and-recover.org/about/ The download contains, at least, borg-example.conf Gerrit A. Smit AT COMPUTING BV From public at enkore.de Fri Jul 28 07:09:45 2017 From: public at enkore.de (Marian Beermann) Date: Fri, 28 Jul 2017 13:09:45 +0200 Subject: [Borgbackup] Bare Metal Backup? In-Reply-To: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> References: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> Message-ID: <934bcff5-4e86-6d3d-daff-536490721083@enkore.de> On 28.07.2017 12:18, Stan Armstrong wrote: > Is there a straightforward way of using borgbackup to do a full metal > > backup/restore of my system partition? > > I am running Debian Stretch 9.1. I would like to back it up in such a > way that if my SSD boot drive is destroyed, I can restore my system > immediately in bootable condition. All of the solutions I have located > are too complex for my aging brain. I have understood borg well enough > to use it for many months, backing up various partitions to and from > other drives both locally and via SSH on my household LAN. I automated > everything by writing simple scripts and using cron. > > I suspect that I simply don't know how to make a restored partition > bootable. > Personally I always found it easiest to just backup system files and re-install the bootloader when the system disk failed. Path of least resistance and all. Cheers, Marian From anarcat at debian.org Fri Jul 28 10:58:09 2017 From: anarcat at debian.org (=?utf-8?Q?Antoine_Beaupr=C3=A9?=) Date: Fri, 28 Jul 2017 10:58:09 -0400 Subject: [Borgbackup] Bare Metal Backup? In-Reply-To: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> References: <25461f2a-75a2-e168-b638-b18a786bde14@gmail.com> Message-ID: <87h8xwpq0u.fsf@curie.anarc.at> On 2017-07-28 07:18:33, Stan Armstrong wrote: > Is there a straightforward way of using borgbackup to do a full metal > > backup/restore of my system partition? > > I am running Debian Stretch 9.1. I would like to back it up in such a > way that if my SSD boot drive is destroyed, I can restore my system > immediately in bootable condition. All of the solutions I have located > are too complex for my aging brain. I have understood borg well enough > to use it for many months, backing up various partitions to and from > other drives both locally and via SSH on my household LAN. I automated > everything by writing simple scripts and using cron. > > I suspect that I simply don't know how to make a restored partition > bootable. I suspect so as well. :) There's nothing like just trying a full restore to see if you actually have a "bare metal" backup. Nothing short of that will prove that you have the correct data set but also procedures. I'm in the process of doing exactly that here, by changing my disk replacement procedure with a full backup/restore procedure. It's a higher downtime than a RAID replacement, of course, but this is a home system, not High Availability (HA) so it seems like the right course of action. My procedure, therefore, is to: 1. format the hard drive (may vary according to your configuration, mine includes LUKS crypto and LVM, for example) 2. make a manual backup backup 3. restore to the hard drive 4. restore boot loader (probably just grub-install in a chroot) 5. reboot on the new drive I still have to test this, and I suspect I'm missing a bunch of things from the backup, but that's the gist of it for now. :) Cheers, A. -- Brief is this existence, as a fleeting visit in a strange house. The path to be pursued is poorly lit by a flickering consciousness. - Albert Einstein From elladan at eskimo.com Thu Aug 3 16:16:12 2017 From: elladan at eskimo.com (Elladan) Date: Thu, 3 Aug 2017 13:16:12 -0700 Subject: [Borgbackup] Deduplication and encryption In-Reply-To: References: Message-ID: On Tue, Jul 18, 2017 at 6:12 AM, Gigi Canuto wrote: > > I am concerned about security and privacy, encryption is a must. Also > because probably I'll need to backup a lot of machines I need deduplication > and (eventually) compression. > I wondered how Borg could handle deduplication and encryption together, so I > asked this question on superuser.com forum. > Could you please confirm that the answer is correct ? The description on that web site is confusing. For more information, see: https://borgbackup.readthedocs.io/en/stable/ So make it simple, the local machine doing the backup does the encryption and deduplication. The remote server machine simply stores the encrypted data but cannot decrypt it. You can back up multiple machines this way, but there are two caveats: First, each local machine has to use the same encryption key, which means that any of the machines can read the other machines' data. Only the remote storage machine is secure. Second, each local machine needs to maintain a cache index to do the deduplication. This is done automatically for you, but takes some extra time to rebuild every time you switch machines. It also takes up some space on each machine. Depending on your needs, it's possible to back up via network file systems and the like which may help in some scenarios. From panayotis at panayotis.com Fri Aug 4 03:00:34 2017 From: panayotis at panayotis.com (Panayotis Katsaloulis) Date: Fri, 4 Aug 2017 10:00:34 +0300 Subject: [Borgbackup] Deduplication and encryption In-Reply-To: References: Message-ID: > > > Second, each local machine needs to maintain a cache index to do the > deduplication. This is done automatically for you, but takes some > extra time to rebuild every time you switch machines. It also takes up > some space on each machine. > > I have seen that too, and for me backup from multiple machines is a requirement. I am using borg this way as a perfect n-way sync tool - even if something goes wrong I can dig to an earlier version. What I've seen and I wish it could be improved, is that every time it seems to bring a lot of meta data, even from those backups that originate from my machine. I was expecting to bring only hashes of the machines in between, not from mines. Let me explain a little bit, if I may. Let's say I have machine A and machine B. So I perform backup A-1, A-2, A3. Then when I try to send backup B-1, before that I get hashes from A-1, A-2, A3 (which makes sense). So the order now of backups is A-1, A-2, A-3, B-1. When I try back to perform backup on A again, then I get hashes from all backups, A-1, A-2, A-3, B-1 , while I was expecting only hasjed from B-1. If I have many backups, which is the case of course, then this gets big really quick... PS: I don't know if it fetches the 100% of older hashes, but for sure it fetches all foreign hashes and most self hashes. -- Panayotis Katsaloulis -------------- next part -------------- An HTML attachment was scrubbed... URL: From borgbackup at aluaces.fastmail.com Sun Aug 13 17:37:01 2017 From: borgbackup at aluaces.fastmail.com (Alberto Luaces) Date: Sun, 13 Aug 2017 23:37:01 +0200 Subject: [Borgbackup] The size of a upgraded Attic repository is almost doubled Message-ID: <1502660221.2514859.1072171584.6B67022B@webmail.messagingengine.com> Hello, while converting an old Attic repository, and after the first Borg backup, I noticed that the size of the whole repository was almost doubled. I suspect that what is happening is that Borg is simply storing the old Attic backups, but since new ones use a different algorithm, they cannot share data blocks, thus the increase in size. I am not particularly worried about this, since I will eventually prune the old backups, but I wanted to know if I am right, in order to account enough space for future upgrades of other Attic repositories I have. Thanks! -- Alberto From carl.beech at gmail.com Mon Aug 14 14:13:47 2017 From: carl.beech at gmail.com (Carl Beech) Date: Tue, 15 Aug 2017 06:13:47 +1200 Subject: [Borgbackup] Compression and De-duplication Message-ID: Hi Apologies if this is answered elsewhere, but I've not seen an answer in the docs, nor a scan of the mailing list archives... I'm new to borg, having moved across from obnam, as maintenance is in the process of stopping - so I'm in the process of setting up a new backup regime. I'm interested to know at what point compression occurs within the backup process - does borg perform a compress on a file, then split into chunks to check for de-duplication, or does it split a file into chunks for deduplication, then compress those chunks when saving them to disk? The reason I ask is that I use hyperconverged storage at work (i.e. storage within our VMWare environment is de-duplicated real-time) - and one of the things pointed out by the vendor is that de-duplication is severely affected by compression - i.e. if you compress a file, then try to de-dup, the de-duplication can't locate similar blocks. For example, if you have a large file, and modify the start of a file, when the compression is performed, the whole file is different from the original (compressed) version. Whereas, leaving the file in an uncompressed state, only the blocks that are changed are stored, and so you end up with a greater compression ratio, even though this appears counter-intuative. A classic case is when you perform a backup of a database file. Hopefully the above makes sense. Many thanks Carl. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Mon Aug 14 16:17:57 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 14 Aug 2017 22:17:57 +0200 Subject: [Borgbackup] The size of a upgraded Attic repository is almost doubled In-Reply-To: <1502660221.2514859.1072171584.6B67022B@webmail.messagingengine.com> References: <1502660221.2514859.1072171584.6B67022B@webmail.messagingengine.com> Message-ID: > while converting an old Attic repository, and after the first Borg > backup, I noticed that the size of the whole repository was almost > doubled. That is expected, search the docs / faq for --chunker-params for an explanation. > I suspect that what is happening is that Borg is simply storing the > old Attic backups, but since new ones use a different algorithm, they > cannot share data blocks, thus the increase in size. Same algorithm, but different parameters. > I am not particularly worried about this, since I will eventually > prune the old backups Yes. And likely you will want to go that route. >, but I wanted to know if I am right, in order to > account enough space for future upgrades of other Attic repositories I > have. You can avoid the repo size doubling by using attic-compatible chunker params. But that will have a negative impact on resource usage then (and long term), so maybe rather don't do that. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Mon Aug 14 16:21:29 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 14 Aug 2017 22:21:29 +0200 Subject: [Borgbackup] Deduplication and encryption In-Reply-To: References: Message-ID: <21a05f7c-e85d-9786-05f2-819568e1d00c@waldmann-edv.de> > What I've seen and I wish it could be improved, is that every time it > seems to bring a lot of meta data, even from those backups that > originate from my machine. > I was expecting to bring only hashes of the machines in between, not > from mines. Yes, that is a bit strange, but it only happens once per archive. Then you'll have a locally cached index of just that archive and it won't fetch it from remote again. It doesn't build that single-archive index when doing the backup (it just updates the all-archives index), was simpler to implement that way. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Mon Aug 14 16:14:14 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 14 Aug 2017 22:14:14 +0200 Subject: [Borgbackup] Compression and De-duplication In-Reply-To: References: Message-ID: <873c0cea-7326-5f69-932c-209ca6553510@waldmann-edv.de> > I'm interested to know at what point compression occurs within the > backup process - does borg perform a compress on a file, then split into > chunks to check for de-duplication, or does it split a file into chunks > for deduplication, then compress those chunks when saving them to disk? borg create: read-from-file and chunk, id-hash, compress, encrypt, add-auth, store-to-repo borg extract: fetch-from-repo, check-auth, decrypt, decompress, write-to-file > The reason I ask is that I use hyperconverged storage at work (i.e. > storage within our VMWare environment is de-duplicated real-time) - and > one of the things pointed out by the vendor is that de-duplication is > severely affected by compression - i.e. if you compress a file, then try > to de-dup, the de-duplication can't locate similar blocks. Yes, that's true for stream compression, like tar.gz. Not so much for "rsyncable" compression. Not sure what's the problem with your storage and borg though. If your source files are on that storage, it won't matter for borg. If your repo is on that storage, your hyperconverged-dedup won't work due to encryption. But it doesn't need to work because a borg repo internally is already deduped (by borg). So, no problem. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From borgbackup at aluaces.fastmail.com Mon Aug 14 17:14:04 2017 From: borgbackup at aluaces.fastmail.com (Alberto Luaces) Date: Mon, 14 Aug 2017 23:14:04 +0200 Subject: [Borgbackup] The size of a upgraded Attic repository is almost doubled In-Reply-To: References: <1502660221.2514859.1072171584.6B67022B@webmail.messagingengine.com> Message-ID: <1502745244.1055913.1073341824.45CB229A@webmail.messagingengine.com> On Mon, Aug 14, 2017, at 10:17 PM, Thomas Waldmann wrote: > > while converting an old Attic repository, and after the first Borg > > backup, I noticed that the size of the whole repository was almost > > doubled. > > That is expected, search the docs / faq for --chunker-params for an > explanation. > > > I suspect that what is happening is that Borg is simply storing the > > old Attic backups, but since new ones use a different algorithm, they > > cannot share data blocks, thus the increase in size. > > Same algorithm, but different parameters. > > > I am not particularly worried about this, since I will eventually > > prune the old backups > > Yes. And likely you will want to go that route. > > >, but I wanted to know if I am right, in order to > > account enough space for future upgrades of other Attic repositories I > > have. > > You can avoid the repo size doubling by using attic-compatible chunker > params. > > But that will have a negative impact on resource usage then (and long > term), so maybe rather don't do that. > Thanks Thomas for your advice; I will follow that route. -- Alberto From carl.beech at gmail.com Tue Aug 15 15:18:30 2017 From: carl.beech at gmail.com (Carl Beech) Date: Wed, 16 Aug 2017 07:18:30 +1200 Subject: [Borgbackup] Compression and De-duplication Message-ID: Many thanks - much appreciated! Carl. > I'm interested to know at what point compression occurs within the > backup process - does borg perform a compress on a file, then split into > chunks to check for de-duplication, or does it split a file into chunks > for deduplication, then compress those chunks when saving them to disk? borg create: read-from-file and chunk, id-hash, compress, encrypt, add-auth, store-to-repo borg extract: fetch-from-repo, check-auth, decrypt, decompress, write-to-file > The reason I ask is that I use hyperconverged storage at work (i.e. > storage within our VMWare environment is de-duplicated real-time) - and > one of the things pointed out by the vendor is that de-duplication is > severely affected by compression - i.e. if you compress a file, then try > to de-dup, the de-duplication can't locate similar blocks. Yes, that's true for stream compression, like tar.gz. Not so much for "rsyncable" compression. Not sure what's the problem with your storage and borg though. If your source files are on that storage, it won't matter for borg. If your repo is on that storage, your hyperconverged-dedup won't work due to encryption. But it doesn't need to work because a borg repo internally is already deduped (by borg). So, no problem. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marioxcc.MT at yandex.com Sun Aug 20 22:17:16 2017 From: marioxcc.MT at yandex.com (Mario =?UTF-8?B?Q2FzdGVsw6Fu?= Castro) Date: Sun, 20 Aug 2017 21:17:16 -0500 Subject: [Borgbackup] =?utf-8?q?Warning=3A_=E2=80=9CUsing_a_pure-python_m?= =?utf-8?q?sgpack!_This_will_result_in_lower_performance=2E=E2=80=9D?= Message-ID: <20170820211716.1c57a4bc@yandex.com> Hello. I am using Debian 9. I am trying to run the latest release tag version of borgbackup from the git repository. When I run ?borg create ...? I get the following message: ?Using a pure-python msgpack! This will result in lower performance.? I have installed all the packages recommended in for Debian, except sphinx and the fuse-specific packages. In addition, I installed the packages "python3-dev". "cython3", "python3-pip" and "python3-setuptools", then did "git clean -xdf" and re-did the procedure (again, as described in the ?Installation? page). I am using virtualenv as the documentation recommends. I also have the ?borgbackup? Debian package installer (which is an older version). If I run Debian's version, it does not display this warning, but if I run it after having sourced the virtualenv script, then it does display the warning. Thus the problem appears related to virtualenv. What can I do to run borg with support for native code within virtualenv (or at least without having to do a system-wide install)? Thanks. -- Do not eat animals, respect them as you respect people. https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 228 bytes Desc: OpenPGP digital signature URL: From tw at waldmann-edv.de Mon Aug 21 00:01:42 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 21 Aug 2017 06:01:42 +0200 Subject: [Borgbackup] =?utf-8?q?Warning=3A_=E2=80=9CUsing_a_pure-python_m?= =?utf-8?q?sgpack!_This_will_result_in_lower_performance=2E=E2=80=9D?= In-Reply-To: <20170820211716.1c57a4bc@yandex.com> References: <20170820211716.1c57a4bc@yandex.com> Message-ID: <68924309-d80b-413d-ab7a-71dcf0f7bc27@waldmann-edv.de> > ?Using a pure-python msgpack! This will result in lower performance.? Usually you get this warning because it could not compile the C++ implementation of msgpack and just falls back to the Python implementation. Often this is due to a missing C++ compiler. So, just check if you can execute: g++ --version That's maybe a helpful command to force the reinstallation of msgpack into your virtualenv: pip install -v --upgrade --force-reinstall --no-cache-dir msgpack-python -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From nickeforos at gmail.com Mon Aug 21 19:12:44 2017 From: nickeforos at gmail.com (Nick Gilmour) Date: Tue, 22 Aug 2017 01:12:44 +0200 Subject: [Borgbackup] Cannot exclude directories Message-ID: Hi all, I cannot exclude any directories with the option **--exclude-from** which I have defined in a text file. The directories in the text file are like this: ... "/home/*/.cache" "*/.Trash-*" ... I have tried some variations of this. E.g.: /.cache /.cache/ .cache .cache/* but Borg keeps creating backups of these directories. Also the option **--exclude-caches** didn't help in the case of .cache. The command I run is of this form: **borg create -spv --exclude-caches --exclude-from exclude-file.txt myrepo::mybackup mysource** borg version: 1.0.7 System: Ubuntu 16.04.1 based What am I doing wrong? Regards, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurice.libes at osupytheas.fr Tue Aug 22 04:27:17 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Tue, 22 Aug 2017 10:27:17 +0200 Subject: [Borgbackup] Cannot exclude directories In-Reply-To: References: Message-ID: Le 22/08/2017 ? 01:12, Nick Gilmour a ?crit : > Hi all, > > I cannot exclude any directories with the option > **--exclude-from** which I have defined in a text file. The > directories in the text file are like this: > ... > "/home/*/.cache" > "*/.Trash-*" > ... > > I have tried some variations of this. E.g.: > /.cache > /.cache/ > .cache > .cache/* > > but Borg keeps creating backups of these directories. Also the option > **--exclude-caches** didn't help in the case of .cache. The command I > run is of this form: > > **borg create -spv --exclude-caches --exclude-from exclude-file.txt > myrepo::mybackup mysource** > > borg version: 1.0.7 > System: Ubuntu 16.04.1 based > > What am I doing wrong? don't know what you're doing wrong but it works for me with /*--exclude '/home/*/.cache' \*/ maybe a syntax problem? borg -V $ borg 1.0.7 * on the mounted backup ll /mnt/provigo-borg/home/libes/.cache ls: impossible d'acc?der ? '/mnt/provigo-borg/home/libes/.cache': Aucun fichier ou dossier de ce type * on my disk ll ~libes/.cache/ -d drwxr-xr-x 18 libes libes 4096 juin 7 12:32 /home/libes/.cache// here is my syntax, with the absolute path $BORG create \ -v --stats \ $COMPRESSION \ $BORG_ARCHIVE \ $REP_ASAUVEGARDER \ /* --exclude '/home/*/.cache' \*/ >> ${LOG_PATH} 2>&1 > > Regards, > Nick > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Tue Aug 22 04:42:02 2017 From: public at enkore.de (Marian Beermann) Date: Tue, 22 Aug 2017 10:42:02 +0200 Subject: [Borgbackup] Cannot exclude directories In-Reply-To: References: Message-ID: Hi Nick, On 22.08.2017 01:12, Nick Gilmour wrote: > Hi all, > > I cannot exclude any directories with the option > **--exclude-from** which I have defined in a text file. The directories > in the text file are like this: > ... > "/home/*/.cache" > "*/.Trash-*" > ... Quoting / escaping doesn't work in exclude files, because Borg never processes quoting / escaping ? it's done by the shell on the command line or in scripts. Cheers, Marian From nickeforos at gmail.com Tue Aug 22 09:32:38 2017 From: nickeforos at gmail.com (Nick Gilmour) Date: Tue, 22 Aug 2017 15:32:38 +0200 Subject: [Borgbackup] Cannot exclude directories In-Reply-To: References: Message-ID: Thanks both! I have tested again like you suggested and it works (without quoting and with paths starting from root). Like this: */dir # excludes directory & all its content */dir/ # excludes only directory's content - keeps directory Regards, Nick On Tue, Aug 22, 2017 at 10:42 AM, Marian Beermann wrote: > Hi Nick, > > On 22.08.2017 01:12, Nick Gilmour wrote: > > Hi all, > > > > I cannot exclude any directories with the option > > **--exclude-from** which I have defined in a text file. The directories > > in the text file are like this: > > ... > > "/home/*/.cache" > > "*/.Trash-*" > > ... > > Quoting / escaping doesn't work in exclude files, because Borg never > processes quoting / escaping ? it's done by the shell on the command > line or in scripts. > > Cheers, Marian > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dave at artlauncher.online Tue Aug 22 13:10:13 2017 From: dave at artlauncher.online (David Thibault) Date: Tue, 22 Aug 2017 10:10:13 -0700 Subject: [Borgbackup] Question about borg remote/local setup Message-ID: Hello, I've got Borg working with all of the files (data, indexes, and everything) on an NFS share (actually, it's Amazon EFS). It's actually backing up image data from one NFS share (my WP Uploads files) to the borg repo on a different NFS share. It works fine, but it takes about 45 minutes to run through less than 3GB of image data, even on subsequent runs. I'm guessing the process would be much faster if I stored the Borg metadata on the local server and only the data blocks on NFS. Then, after the main borg backup, I could do a cron script that would zip up the remaining borg metadata files, then borg could back that zip file as well. Thoughts? It seems like it would be much better performance than storing everything on NFS. If you agree, which files would be best kept on the local disk and which should be on NFS. Judging by the repo contents, I'm guessing the data folder on NFS and the rest on the local disk. Can you confirm? Thanks, Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Tue Aug 22 13:37:36 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 22 Aug 2017 19:37:36 +0200 Subject: [Borgbackup] Question about borg remote/local setup In-Reply-To: References: Message-ID: <88255ba0-4a84-d63e-c1b3-edf4b11aa2b6@waldmann-edv.de> > I've got Borg working with all of the files (data, indexes, and > everything) on an NFS share (actually, it's Amazon EFS). It's actually > backing up image data from one NFS share (my WP Uploads files) to the > borg repo on a different NFS share. > > It works fine, but it takes about 45 minutes to run through less than > 3GB of image data, even on subsequent runs. That's likely because it reads/chunks everything because it thinks the file has changed. This is when one of mtime, inode nr, size is not stable (usually the inode nr is not stable on network filesystems). If you can trust only mtime and size for the "is the file changed" detection, you can tell borg to ignore the inode number (see --ignore-inode or so). > I'm guessing the process > would be much faster if I stored the Borg metadata on the local server > and only the data blocks on NFS. Then, after the main borg backup, I > could do a cron script that would zip up the remaining borg metadata > files, then borg could back that zip file as well. Not sure how you mean that, but in general local storage is usually faster than network storage. Also have a look at cache / index sizes and maybe increase borg 1.0.x --checkpoint-interval from 300 (5mins) to 1800 (30mins) to have fewer checkpoints but also less overhead - esp. if indexes/caches are large and reading/writing them is slow. > If you agree, which files would be best kept on the local disk and which > should be on NFS. Judging by the repo contents, I'm guessing the data > folder on NFS and the rest on the local disk. Can you confirm? Maybe try the --checkpoint-interval first, then you maybe do not have to do that. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From giuseppe.arvati at gmail.com Fri Aug 25 06:02:08 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Fri, 25 Aug 2017 12:02:08 +0200 Subject: [Borgbackup] performance problem Message-ID: Hello, I?m trying to use borg as backup tool for a file server but I?m not able to finish the backup during the night. The server is an Proliant 8 core Intel(R) Xeon(R) E5540 @ 2.53GHz with raid 5 scsi controller. Borg version is 1.0.11 I have mainly 3 directory to back up: 184G /dati/utenti 354G /dati/gruppi 90G /dati/passa and I would to backup all three directory every day between 23:00 and 4:00 ( but it can be 22:00 ? 5:00 ). Now I use rsync to copy new o modified files on NAS and it take about 30 minutes to complete but I do not have an history and a deduplication. The first directory one take about 11 hours to backup 184GB so is not possible to complete all 3 directory during the night. I tried first to backup on NAS connected via 1Gb NSF link without compression and encryption and I create 2 archive: utenti-2017-07-06 Thu, 2017-07-06 19:00:01 utenti-2017-07-19 Wed, 2017-07-19 15:03:46 the first archive took about 14 hours to finish the second archive took about 9 hours to finish too slow!! So I thought was a link performance problem and I tried again on local storage but without important improvement. Command line: /usr/bin/borg create -v --stat ::utenti-{now:%Y-%m-%d} /dati/utenti utenti-2017-07-25 Tue, 2017-07-25 13:05:00 utenti-2017-07-26 Wed, 2017-07-26 19:00:00 utenti-2017-08-24 Thu, 2017-08-24 13:00:00 the 1st archive took about 12 hours to finish the 2nd archive took about 11 hours to finish the 3rd archive took about 11 hours to finish with with ?ignore-inode too slow again !! 185 GB of data aren?t a big archive and I think borg generally works faster than I experienced. Where are my mistakes ? Missing options, error during archive init phase, server performance, ? What can I test ? Thank you for any hints Giuseppe From tw at waldmann-edv.de Fri Aug 25 08:46:26 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 25 Aug 2017 14:46:26 +0200 Subject: [Borgbackup] performance problem In-Reply-To: References: Message-ID: > I?m trying to use borg as backup tool for a file server but I?m not able > to finish the backup during the night. No problem, just finish it somehow, even if it runs over the day. Maybe your first backup will not be super consistent (as at daytime, people will change files). But it will transfer most data chunks to the repository and create a files cache, so it can be faster next time. Keep the first backup at least until you finished the 2nd backup of the same data. > I would to backup all three directory every day between 23:00 and > 4:00 ( but it can be 22:00 ? 5:00 ). It might take longer the first time, but (assuming that most of your data is not changing and also the files cache of borg is working normally), your subsequent backups will be way faster than the first one. > The first directory one take about 11 hours to backup 184GB so is not > possible to complete all 3 directory during the night. It is, just not the first time. > Where are my mistakes ? Missing options, error during archive init > phase, server performance, ? > What can I test ? Unless your 2nd backup is also unexpectedly slow, there is no big problem. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From marioxcc.MT at yandex.com Fri Aug 25 16:14:18 2017 From: marioxcc.MT at yandex.com (=?UTF-8?Q?Mario_Castel=c3=a1n_Castro?=) Date: Fri, 25 Aug 2017 15:14:18 -0500 Subject: [Borgbackup] =?utf-8?q?Warning=3A_=E2=80=9CUsing_a_pure-python_m?= =?utf-8?q?sgpack!_This_will_result_in_lower_performance=2E=E2=80=9D?= In-Reply-To: <68924309-d80b-413d-ab7a-71dcf0f7bc27@waldmann-edv.de> References: <20170820211716.1c57a4bc@yandex.com> <68924309-d80b-413d-ab7a-71dcf0f7bc27@waldmann-edv.de> Message-ID: <70ca1aa2-406c-5bdf-0af1-9f0f6413eed9@yandex.com> On 20/08/17 23:01, Thomas Waldmann wrote: >> ?Using a pure-python msgpack! This will result in lower performance.? > > Usually you get this warning because it could not compile the C++ > implementation of msgpack and just falls back to the Python > implementation. Often this is due to a missing C++ compiler. > > So, just check if you can execute: g++ --version Thanks for the reply and sorry for the delay. I had g++ installed already. I suspect that the problem is that virtualenv isn't copying the binary files (?.so?) to the virtual environment, and therefore the native implementation is missing. Anyway, I found that using borgbackup from the upstream source requires a significant amount of configuration for its environment. Since I want something that works ?out of the box? I still use borgbackup, but the version packaged by Debian instead. Regards. -- Do not eat animals, respect them as you respect people. https://duckduckgo.com/?q=how+to+(become+OR+eat)+vegan -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 228 bytes Desc: OpenPGP digital signature URL: From nickeforos at gmail.com Sat Aug 26 10:21:36 2017 From: nickeforos at gmail.com (Nick Gilmour) Date: Sat, 26 Aug 2017 16:21:36 +0200 Subject: [Borgbackup] How to list and delete all repos? Message-ID: Hi all, I have created on my server some test repositories and now I want to delete them. 1. Is there any command to list all existing repos under a certain path e.g. root. I could of course locate them with other means but I mean does borg have something included? 2. Assuming I know where they are, is there a command to delete them all? What if I simply delete the folders where they reside without using the borg delete command? Regards Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Sat Aug 26 22:48:19 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 27 Aug 2017 04:48:19 +0200 Subject: [Borgbackup] How to list and delete all repos? In-Reply-To: References: Message-ID: <64dbc98e-9f88-2625-dcda-f3c7a0efc3b6@waldmann-edv.de> > I have created on my server some test repositories and now I want to > delete them. > > 1. Is there any command to list all existing repos under a certain path > e.g. root. There is no borg command for that. > 2. Assuming I know where they are, is there a command to delete them > all? Only one-by-one: borg delete > What if I simply delete the folders where they reside without using > the borg delete command? You can use rm -rf to get rid of the repo, but that would not delete the clientside cache. When you use borg, it will also delete the clientside cache for that repo, IF you run the delete command on the correct client. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Sun Aug 27 20:50:49 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 28 Aug 2017 02:50:49 +0200 Subject: [Borgbackup] borgbackup 1.1.0 release candidate 2 released Message-ID: <79f626a7-7e7b-d7f0-4583-77d786679c62@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.1.0rc2 please help testing and provide feedback. details see url. cheers, thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From eric at in3x.io Mon Aug 28 01:13:20 2017 From: eric at in3x.io (Eric S. Johansson) Date: Mon, 28 Aug 2017 01:13:20 -0400 Subject: [Borgbackup] Index object count mismatch, how to fix Message-ID: <93eb600d-e099-8ef0-41df-4009012e545c@in3x.io> I started getting backup outputs with cost exceptions and I thought I just had another version mismatch. I upgraded both sides so they with the same version of borg (1.0.11) on both sides and then ran a repository check. I got the index object count mismatch error. I then ran the same check on other archives I've made and they also have index object count mismatches. On my own personal archive I'm running check with the repair option and I'm getting a lot of messages that say: np1/pond/home/eric/windows-jun-25-2013.tar.gz: New missing file chunk detected (Byte 66560501323-66562026633). Replacing with all-zero chunk. I did a bit of googling and I turned up a bug report from about a year ago which says the index mismatch is a nonissue and that he better healing was implemented in item 1300. My interpretation of what I read is that I need to do repair and then do a new backup in order to get rid of the problems. Is that the right plan?? I'll make sure I do the right recovery because I can accept toasting my own data but would really hate to mess it up for anybody else. --- eric -- Eric S. Johansson eric at in3x.io http://www.in3x.io 978-512-0272 From eric at in3x.io Mon Aug 28 01:29:59 2017 From: eric at in3x.io (Eric S. Johansson) Date: Mon, 28 Aug 2017 01:29:59 -0400 Subject: [Borgbackup] Index object count mismatch, how to fix In-Reply-To: <93eb600d-e099-8ef0-41df-4009012e545c@in3x.io> References: <93eb600d-e099-8ef0-41df-4009012e545c@in3x.io> Message-ID: <1c83e4af-db90-8a03-b2ae-c781024361b8@in3x.io> one more oddity. looking at the repair output, I found: np1/pond/home/eric/windows-jun-25-2013.tar.gz: New missing file chunk detected (Byte 103088780039-103090572092). Replacing with all-zero chunk. np1/pond/home/eric/windows-jun-25-2013.tar.gz: New missing file chunk detected (Byte 103090572092-103091532307). Replacing with all-zero chunk. np1/pond/home/eric/windows-jun-25-2013.tar.gz: New missing file chunk detected (Byte 103091532307-103092527022). Replacing with all-zero chunk. the missing chunks are sequential. does that mean anything? --- eric -- Eric S. Johansson eric at in3x.io http://www.in3x.io 978-512-0272 From bioc0765 at outlook.com Mon Aug 28 11:18:14 2017 From: bioc0765 at outlook.com (Judith Brown) Date: Mon, 28 Aug 2017 15:18:14 +0000 Subject: [Borgbackup] suggestion append data for existing archive In-Reply-To: References: Message-ID: If am searching for a backup + optimal solution to raspberrypi media management. Assume I have a (2TB) large /media/SOURCE and small /home/pi (~8GB from microSDcard). I do a first backup: borg init /media/2TB_BACKUP borg create? /media/2TB_BACKUP::archive-1? /media/SOURCE Now, I mount borg mount? /media/2TB_BACKUP::archive-1? /media/DATA I remove 2TB /media/SOURCE Now I mount a small 8GB usb stick at /media/SOURCE My suggestions: 1. If I run (with 8GB usb stick) borg create? /media/2TB_BACKUP::archive-2? /media/SOURCE this archive will have only the NEW 8GB data. 2. Can I access *all* data from archive-1 + archive-2 at the sametime at the same mountpoint? Instead of (1), it would be great if there was a 'append' parameter, so that I can 'append' data onto the archive-1 thereby while 'creating+adding new data borg does compression+dedup. This after initial setup I can make /media/2TB_BACKUP and make it my main 'disk' since it features deduplication and compression! (onto any filesystem - like ext4 or XFS). I wouldn't mind umounting eachtime before running a 'append'. Do you think this qualifies as a feature for future? Thanks for the great work. From public at enkore.de Mon Aug 28 12:03:34 2017 From: public at enkore.de (Marian Beermann) Date: Mon, 28 Aug 2017 18:03:34 +0200 Subject: [Borgbackup] suggestion append data for existing archive In-Reply-To: References: Message-ID: <32e4b770-8c5e-0c77-6090-ad505927333f@enkore.de> Hi Judith, On 28.08.2017 17:18, Judith Brown wrote: > If am searching for a backup + optimal solution to raspberrypi media management. > Assume I have a (2TB) large /media/SOURCE and small /home/pi (~8GB from microSDcard). > I do a first backup: > borg init /media/2TB_BACKUP > borg create /media/2TB_BACKUP::archive-1 /media/SOURCE > > Now, I mount > borg mount /media/2TB_BACKUP::archive-1 /media/DATA > > I remove 2TB /media/SOURCE > Now I mount a small 8GB usb stick at /media/SOURCE > > My suggestions: > 1. If I run (with 8GB usb stick) > borg create /media/2TB_BACKUP::archive-2 /media/SOURCE > this archive will have only the NEW 8GB data. Yes > 2. Can I access *all* data from archive-1 + archive-2 at the sametime at the same mountpoint? You can mount the entire repository (just leave out the "::archive" bit). Each archive is then a directory under the mountpoint. (Because archives are loaded lazily this doesn't consume many resources for many archives at first). > Instead of (1), it would be great if there was a 'append' parameter, > so that I can 'append' data onto the archive-1 thereby while > 'creating+adding new data borg does compression+dedup. This after > initial setup I can make /media/2TB_BACKUP and make it my main > 'disk' since it features deduplication and compression! (onto any > filesystem - like ext4 or XFS). An "append-to-archive" operation has been brought up before but is usually not considered to be in the spirit of Borg. See e.g. https://github.com/borgbackup/borg/issues/2143 https://github.com/borgbackup/borg/issues/1935 I also once had the idea to use Borg for media archival / management as well (strong checksumming, dedup), but it's not really practical with the current features and the necessary extensions to enable this use were not considered a good fit by project members. See https://github.com/borgbackup/borg/issues/2263 I settled on ZFS (zfs on linux), btw. Cheers, Marian From bioc0765 at outlook.com Mon Aug 28 08:13:55 2017 From: bioc0765 at outlook.com (Judith Brown) Date: Mon, 28 Aug 2017 12:13:55 +0000 Subject: [Borgbackup] suggestion append data for existing archive Message-ID: If am searching for a backup + optimal solution to raspberrypi media management. Assume I have a (2TB) large /media/SOURCE and small /home/pi (~8GB from microSDcard). I do a first backup: borg init /media/2TB_BACKUP borg create /media/2TB_BACKUP::archive-1 /media/SOURCE Now, I mount borg mount /media/2TB_BACKUP::archive-1 /media/DATA I remove 2TB /media/SOURCE Now I mount a small 8GB usb stick at /media/SOURCE My suggestions: 1. If I run (with 8GB usb stick) borg create /media/2TB_BACKUP::archive-2 /media/SOURCE this archive will have only the NEW 8GB data. 2. Can I access *all* data from archive-1 + archive-2 at the sametime at the same mountpoint? Instead of (1), it would be great if there was a 'append' parameter, so that I can 'append' data onto the archive-1 thereby while 'creating+adding new data borg does compression+dedup. This after initial setup I can make /media/2TB_BACKUP and make it my main 'disk' since it features deduplication and compression! (onto any filesystem - like ext4 or XFS). I wouldn't mind umounting eachtime before running a 'append'. Do you think this qualifies as a feature for future? Thanks for the great work. From giuseppe.arvati at gmail.com Thu Aug 31 05:44:45 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Thu, 31 Aug 2017 11:44:45 +0200 Subject: [Borgbackup] performance problem In-Reply-To: References: Message-ID: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> Hello, again on performance the 1st archive is about 195GB with 462.000 files and take about 11 hours to complete the 2nd archive is about 377GB with 1.020.000 files and take less then 2 hours I do not understand what can slow down the performance of the first archive. Which tests can I do to debug the first archive ? Thank you Archive name: utenti-2017-08-30-19 Time (start): Wed, 2017-08-30 19:30:18 Time (end): Thu, 2017-08-31 06:40:27 Duration: 11 hours 10 minutes 8.99 seconds Number of files: 462310 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 194.81 GB 194.83 GB 35.63 MB All archives: 584.40 GB 584.46 GB 156.92 GB Unique chunks Total chunks Chunk index: 378370 1481376 ------------------------------------------------------------------------------ ------------------------------------------------------------------------------ Archive name: gruppi-2017-08-30-19 Time (start): Thu, 2017-08-31 06:40:36 Time (end): Thu, 2017-08-31 08:19:50 Duration: 1 hours 39 minutes 13.62 seconds Number of files: 1021125 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 376.39 GB 376.44 GB 161.53 MB All archives: 1.13 TB 1.13 TB 315.65 GB Unique chunks Total chunks Chunk index: 948040 3423105 Il 25/08/2017 14:46, Thomas Waldmann ha scritto: >> I?m trying to use borg as backup tool for a file server but I?m not able >> to finish the backup during the night. > > No problem, just finish it somehow, even if it runs over the day. > > Maybe your first backup will not be super consistent (as at daytime, > people will change files). But it will transfer most data chunks to the > repository and create a files cache, so it can be faster next time. > > Keep the first backup at least until you finished the 2nd backup of the > same data. > >> I would to backup all three directory every day between 23:00 and >> 4:00 ( but it can be 22:00 ? 5:00 ). > > It might take longer the first time, but (assuming that most of your > data is not changing and also the files cache of borg is working > normally), your subsequent backups will be way faster than the first one. > >> The first directory one take about 11 hours to backup 184GB so is not >> possible to complete all 3 directory during the night. > > It is, just not the first time. > >> Where are my mistakes ? Missing options, error during archive init >> phase, server performance, ? >> What can I test ? > > Unless your 2nd backup is also unexpectedly slow, there is no big problem. > > > From tw at waldmann-edv.de Thu Aug 31 08:31:51 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 31 Aug 2017 14:31:51 +0200 Subject: [Borgbackup] performance problem In-Reply-To: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> References: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> Message-ID: <39c2bd07-820f-fd14-4327-438c28542ce2@waldmann-edv.de> > the 1st archive is about 195GB with 462.000 files and take > about 11 hours to complete > > the 2nd archive is about 377GB with 1.020.000 files and take > less then 2 hours Let's rather refer to this as the utenti and gruppi data set. "1st" and "2nd archive" gives the wrong impression that 1st = initial backup, 2nd = subsequent backup (of same data set). > I do not understand what can slow down the performance of the first > archive. Are both backups running on same machine? Are both backups storing into the same repository? Or at least into a repository located on the same borg repo server? The data set read by the backups - where does the data come from? Local fs? nfs? smbfs? other? Different performance might just come from differently performing source file systems. If it is not a local filesystem, lower performance is expected. Do you always backup the same path or do you mount at different places? Pathes need to be stable or the files cache can't work. Also check whether inode numbers are stable. Network filesystems often do not have stable inode numbers, you can have a look at --ignore-inode option of borg create then. Also, I think you should use lz4 compression at least. It usually saves more time than it needs. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From giuseppe.arvati at gmail.com Thu Aug 31 10:03:40 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Thu, 31 Aug 2017 16:03:40 +0200 Subject: [Borgbackup] performance problem In-Reply-To: <39c2bd07-820f-fd14-4327-438c28542ce2@waldmann-edv.de> References: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> <39c2bd07-820f-fd14-4327-438c28542ce2@waldmann-edv.de> Message-ID: <69c7e572-61a0-67c5-4c54-b0b9a04fb97c@gmail.com> Il 31/08/2017 14:31, Thomas Waldmann ha scritto: >> the 1st archive is about 195GB with 462.000 files and take >> about 11 hours to complete >> >> the 2nd archive is about 377GB with 1.020.000 files and take >> less then 2 hours > > Let's rather refer to this as the utenti and gruppi data set. > > "1st" and "2nd archive" gives the wrong impression that 1st = initial > backup, 2nd = subsequent backup (of same data set). > >> I do not understand what can slow down the performance of the first >> archive. > > Are both backups running on same machine? yes in the same job the 2nd start when 1st finish > Are both backups storing into the same repository? Or at least into a > repository located on the same borg repo server? yes to network storage via NSF > The data set read by the backups - where does the data come from? local FS but are two differente samba network share > Different performance might just come from differently performing source > file systems. > > If it is not a local filesystem, lower performance is expected. > Do you always backup the same path or do you mount at different places? > Pathes need to be stable or the files cache can't work. same path ( smb network share ) > Also check whether inode numbers are stable. Network filesystems often > do not have stable inode numbers, you can have a look at --ignore-inode > option of borg create then. I will try > Also, I think you should use lz4 compression at least. It usually saves > more time than it needs. I'll try can be a problem related to lock/timeout for same files served by samba ? thank you Giuseppe From tw at waldmann-edv.de Thu Aug 31 12:59:07 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 31 Aug 2017 18:59:07 +0200 Subject: [Borgbackup] performance problem In-Reply-To: <69c7e572-61a0-67c5-4c54-b0b9a04fb97c@gmail.com> References: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> <39c2bd07-820f-fd14-4327-438c28542ce2@waldmann-edv.de> <69c7e572-61a0-67c5-4c54-b0b9a04fb97c@gmail.com> Message-ID: <219b9977-37ba-d122-59c4-a9f15c76862f@waldmann-edv.de> >> Are both backups storing into the same repository? Or at least into a >> repository located on the same borg repo server? > > yes to network storage via NSF You mean to NFS? This is likely slower than using ssh: repo url (requires borg on the repo server). If your network connections are stable, you can reduce the impact a bit by using --checkpoint-interval=1800 (or more, default is 300s in borg 1.0.x). >> The data set read by the backups - where does the data come from? > local FS but are two differente samba network share OK, if borg is reading source data from physical, local disk, there should be no problem with inode numbers. What's the filesystem type (see mount output)? > can be a problem related to lock/timeout for same files served by samba ? Not sure about that. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From giuseppe.arvati at gmail.com Fri Sep 1 06:13:53 2017 From: giuseppe.arvati at gmail.com (Giuseppe Arvati) Date: Fri, 1 Sep 2017 12:13:53 +0200 Subject: [Borgbackup] performance problem In-Reply-To: <219b9977-37ba-d122-59c4-a9f15c76862f@waldmann-edv.de> References: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> <39c2bd07-820f-fd14-4327-438c28542ce2@waldmann-edv.de> <69c7e572-61a0-67c5-4c54-b0b9a04fb97c@gmail.com> <219b9977-37ba-d122-59c4-a9f15c76862f@waldmann-edv.de> Message-ID: Il 31/08/2017 18:59, Thomas Waldmann ha scritto: >>> Are both backups storing into the same repository? Or at least into a >>> repository located on the same borg repo server? >> >> yes to network storage via NSF > > You mean to NFS? yes > This is likely slower than using ssh: repo url (requires borg on the > repo server). > > If your network connections are stable, you can reduce the impact a bit > by using --checkpoint-interval=1800 (or more, default is 300s in borg > 1.0.x). usually stable ( 1Gb link ) > >>> The data set read by the backups - where does the data come from? >> local FS but are two differente samba network share > > OK, if borg is reading source data from physical, local disk, there > should be no problem with inode numbers. > > What's the filesystem type (see mount output)? > xfs ( /dev/sdb on /dati type xfs (rw,noatime,nodiratime) ) >> can be a problem related to lock/timeout for same files served by samba ? > > Not sure about that. this night I ran the same job through CRON instead of AT with this stats: /dati/utenti Archive name: utenti-2017-08-31-19 Time (start): Thu, 2017-08-31 19:32:51 Time (end): Fri, 2017-09-01 06:45:47 Duration: 11 hours 12 minutes 55.85 seconds Number of files: 462.364 ( 194GB ) /dati/gruppi Archive name: gruppi-2017-08-31-19 Time (start): Fri, 2017-09-01 06:45:56 Time (end): Fri, 2017-09-01 10:56:12 Duration: 4 hours 10 minutes 16.21 seconds Number of files: 1.022.702 ( 380GB ) and I had this new messages Synchronizing chunks cache... Archives: 3, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 3. Fetching and building archive index for utenti-2017-08-30-19 ... Merging into master chunks index ... Fetching and building archive index for utenti-2017-08-28 ... Merging into master chunks index ... Fetching and building archive index for utenti-17-08-29-19 ... Merging into master chunks index ... Are these messages ok ? So I have two archive one is an half of the other and take 3 times to complete the backup. If someone have other suggestions, to try to investigate why the first archive is slower than second archive, will be very appreciate. thank you giuseppe From tw at waldmann-edv.de Fri Sep 1 10:49:54 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 1 Sep 2017 16:49:54 +0200 Subject: [Borgbackup] performance problem In-Reply-To: References: <48561fcc-ff39-f98b-748f-b9441fb30f1d@gmail.com> <39c2bd07-820f-fd14-4327-438c28542ce2@waldmann-edv.de> <69c7e572-61a0-67c5-4c54-b0b9a04fb97c@gmail.com> <219b9977-37ba-d122-59c4-a9f15c76862f@waldmann-edv.de> Message-ID: >> What's the filesystem type (see mount output)? > xfs ( /dev/sdb on /dati type xfs (rw,noatime,nodiratime) ) OK, so there should be stable inode numbers. Also, you will have good metadata stream duplication as the atimes are not changing. > and I had this new messages > > Synchronizing chunks cache... > Archives: 3, w/ cached Idx: 0, w/ outdated Idx: 0, w/o cached Idx: 3. > Fetching and building archive index for utenti-2017-08-30-19 ... > Merging into master chunks index ... > Fetching and building archive index for utenti-2017-08-28 ... > Merging into master chunks index ... > Fetching and building archive index for utenti-17-08-29-19 ... > Merging into master chunks index ... > > Are these messages ok ? The chunks cache needs to get resynchronized if another client is writing to same repository, causing a chunks cache incoherency. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Sat Sep 9 21:34:05 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 10 Sep 2017 03:34:05 +0200 Subject: [Borgbackup] borgbackup 1.1.0 release candidate 3 released Message-ID: https://github.com/borgbackup/borg/releases/tag/1.1.0rc3 please help testing and provide feedback. details see url. cheers, thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From _ at thomaslevine.com Mon Sep 11 06:24:51 2017 From: _ at thomaslevine.com (Thomas Levine) Date: Mon, 11 Sep 2017 10:24:51 +0000 Subject: [Borgbackup] Lots of files that change rarely and predictably Message-ID: <20170911102459.00CD27E850@mailuser.nyi.internal> I have a directory of about 670,683 files in a single borg repository. Archive creation takes a long time because borg checks all of the files. I should be able to make the backup go faster because the files change very predictably, and I seek your advice on how to do this. The directory contains emails in mail handler format, with one email per file. The subdirectories look like this, "->" indicating symbolic links. old/2006 old/2007 old/2008 old/2009 old/2010 old/2011 old/2012 old/2013 old/2014 old/2015 old/2016 old/2017 drafts old/current -> old/2017 inbox -> old/current/inbox sent -> old/current/sent (It is in fact more complicated, but this gets the point across.) Usually, only the files in "drafts" and the target of "old/current" change. I want to configure the backups to take advantage of this pattern, by having borg ignore the files that I know have not changed. The only way I know to do this is to separate repositories. For example, I could use a separate repository for each new "old" directory and create new borg archives only for that directory in that repository. Are there other safe ways to instruct borg to consider changes only to particular files? I think it would work to check the files cache only for the files in the target of "old/current", treating all other indexed files as unchanged. But I'm not sure, and it does not seem to be supported at the moment. Thank you From tw at waldmann-edv.de Mon Sep 11 07:57:21 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 11 Sep 2017 13:57:21 +0200 Subject: [Borgbackup] Lots of files that change rarely and predictably In-Reply-To: <20170911102459.00CD27E850@mailuser.nyi.internal> References: <20170911102459.00CD27E850@mailuser.nyi.internal> Message-ID: <41e96e6b-bd08-aff8-5e6c-07f047d53524@waldmann-edv.de> On 09/11/2017 12:24 PM, Thomas Levine wrote: > I have a directory of about 670,683 files in a single borg repository. > Archive creation takes a long time because borg checks all of the files. > I should be able to make the backup go faster because the files change > very predictably, and I seek your advice on how to do this. Well, if these are years of email, maybe creating an archive of the past years in mbox format would be the easiest performance fix. > > The directory contains emails in mail handler format, with one email per > file. This is kind of known/expected to be slow and inefficient. A filesystem that supports fast handling of such huge directories might help a bit. See dir_index option and similar. > Usually, only the files in "drafts" and the target of "old/current" > change. I want to configure the backups to take advantage of this > pattern, by having borg ignore the files that I know have not changed. I am not sure this is doable. You'ld still have to look into the directory for new files. Borg's files cache lookup only needs to know mtime, size and inode number to decide which files did not change. BTW, can you do a few runs with borg 1.1.0rc3 into a new repo to see whether performance changes when going from 1.0 to 1.1? -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From gmatht at gmail.com Mon Sep 11 08:25:44 2017 From: gmatht at gmail.com (John McCabe-Dansted) Date: Mon, 11 Sep 2017 20:25:44 +0800 Subject: [Borgbackup] Lots of files that change rarely and predictably In-Reply-To: <20170911102459.00CD27E850@mailuser.nyi.internal> References: <20170911102459.00CD27E850@mailuser.nyi.internal> Message-ID: Have you considered using `mount --bind` to hide the files that haven't been changed? For example: mkdir backup mount --bind . backup mkdir tmp/empty mount --bind tmp/empty backup/old/ ... if [ `date +%u` == 6 ] then borg create '/path/to/repo::Weekly-{now:%Y-%m-%d}' . else borg create '/path/to/repo::Daily-{now:%Y-%m-%d}' backup fi Clearly restoring would involve restoring a Daily and then copying the old files from the last Weekly On 11 September 2017 at 18:24, Thomas Levine <_ at thomaslevine.com> wrote: > I have a directory of about 670,683 files in a single borg repository. > Archive creation takes a long time because borg checks all of the files. > I should be able to make the backup go faster because the files change > very predictably, and I seek your advice on how to do this. > > The directory contains emails in mail handler format, with one email per > file. The subdirectories look like this, "->" indicating symbolic links. > > old/2006 > old/2007 > old/2008 > old/2009 > old/2010 > old/2011 > old/2012 > old/2013 > old/2014 > old/2015 > old/2016 > old/2017 > drafts > old/current -> old/2017 > inbox -> old/current/inbox > sent -> old/current/sent > > (It is in fact more complicated, but this gets the point across.) > > Usually, only the files in "drafts" and the target of "old/current" > change. I want to configure the backups to take advantage of this > pattern, by having borg ignore the files that I know have not changed. > > The only way I know to do this is to separate repositories. For example, > I could use a separate repository for each new "old" directory and > create new borg archives only for that directory in that repository. > Are there other safe ways to instruct borg to consider changes only to > particular files? I think it would work to check the files cache only > for the files in the target of "old/current", treating all other indexed > files as unchanged. But I'm not sure, and it does not seem to be > supported at the moment. > > Thank you > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- John C. McCabe-Dansted -------------- next part -------------- An HTML attachment was scrubbed... URL: From jdc at uwo.ca Mon Sep 11 08:59:26 2017 From: jdc at uwo.ca (Dan Christensen) Date: Mon, 11 Sep 2017 08:59:26 -0400 Subject: [Borgbackup] Lots of files that change rarely and predictably In-Reply-To: <20170911102459.00CD27E850@mailuser.nyi.internal> (Thomas Levine's message of "Mon, 11 Sep 2017 10:24:51 +0000") References: <20170911102459.00CD27E850@mailuser.nyi.internal> Message-ID: <87poaxcrwx.fsf@uwo.ca> On Sep 11, 2017, Thomas Levine <_ at thomaslevine.com> wrote: > I have a directory of about 670,683 files in a single borg repository. > Archive creation takes a long time because borg checks all of the files. > I should be able to make the backup go faster because the files change > very predictably, and I seek your advice on how to do this. One repository can contain archives with different content. So you could just run borg create old/2017 for now, and switch to old/2018 next year, etc. You can mix in periodic backups of everything, just to be safe, and borg will correctly dedupicate the content shared between different archives. Dan From jdc at uwo.ca Mon Sep 11 12:00:58 2017 From: jdc at uwo.ca (Dan Christensen) Date: Mon, 11 Sep 2017 12:00:58 -0400 Subject: [Borgbackup] Lots of files that change rarely and predictably In-Reply-To: <20170911102459.00CD27E850@mailuser.nyi.internal> (Thomas Levine's message of "Mon, 11 Sep 2017 10:24:51 +0000") References: <20170911102459.00CD27E850@mailuser.nyi.internal> Message-ID: <87fubt6x8l.fsf@uwo.ca> On Sep 11, 2017, Thomas Levine <_ at thomaslevine.com> wrote: > I have a directory of about 670,683 files in a single borg repository. > Archive creation takes a long time because borg checks all of the files. Another point I should make is that I back up that many files regularly, most of which haven't changed, and it only takes a 2-3 minutes. Are you running on a filesystem with stable inodes? Borg should quickly check mtime, size and inode number, and skip files that it has already backed up. Dan From melkor.lord at gmail.com Mon Sep 11 22:12:52 2017 From: melkor.lord at gmail.com (Melkor Lord) Date: Tue, 12 Sep 2017 04:12:52 +0200 Subject: [Borgbackup] Test : Borg vs Restic Message-ID: Hi, I'm evaluating the right backup service for my needs. This boils down to comparing Restic and Borg as they represent the "top of the shelf" solutions currently available on Linux. Borg : 1.1.0rc3 (from the borg-linux64 binary) Restic : 0.7.1 (from the restic linux_amd64 binary) The backup data consists of a live mail repository using the maildir format and holding 139 GB (2327 dirs, 665456 files). Keep in mind that this is the result of my own experience, for my own needs and this is in no way thorough nor exhaustive. BORG : ====== Note: Encryption is "repokey-blake2" * First pass Shell# time ./borg-linux64 create --info --stats --progress /path/to/BackupTests/Borg::{now:%Y%m%d-%H%M%S} /path/to/Mail/ Enter passphrase for key /path/to/BackupTests/Borg: ------------------------------------------------------------------------------ Archive name: 20170911-164308 Archive fingerprint: ea043fb5154c60ecdcb42e3be238cfa2ad040e03349f5ae5cab6a9f9f8fd48fe Time (start): Mon, 2017-09-11 16:43:11 Time (end): Mon, 2017-09-11 17:58:32 Duration: 1 hours 15 minutes 20.67 seconds Number of files: 646835 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 137.47 GB 112.55 GB 102.30 GB All archives: 137.47 GB 112.55 GB 102.30 GB Unique chunks Total chunks Chunk index: 639574 680719 ------------------------------------------------------------------------------ real 75m33.037s user 23m22.756s sys 3m51.228s * Second pass Shell# time ./borg-linux64 create --info --stats --progress /path/to/BackupTests/Borg::{now:%Y%m%d-%H%M%S} /path/to/Mail/ Enter passphrase for key /path/to/BackupTests/Borg: ------------------------------------------------------------------------------ Archive name: 20170911-181622 Archive fingerprint: 67e9f0e14fa092d274e99833806ca789eb88df890190ec37cedb5e4af20107a0 Time (start): Mon, 2017-09-11 18:16:25 Time (end): Mon, 2017-09-11 18:18:27 Duration: 2 minutes 1.73 seconds Number of files: 646861 Utilization of max. archive size: 0% ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 137.47 GB 112.55 GB 14.57 MB All archives: 274.94 GB 225.10 GB 102.32 GB Unique chunks Total chunks Chunk index: 639728 1361461 ------------------------------------------------------------------------------ real 2m13.070s user 1m55.448s sys 0m12.652s RESTIC : ======== * Fist pass Shell# time ./restic_0.7.1_linux_amd64 backup -r /path/to/BackupTests/Restic/ /path/to/Mail/ enter password for repository: scan [/path/to/Mail] scanned 2327 directories, 665464 files in 0:02 [1:48:16] 100.00% 21.440 MiB/s 136.009 GiB / 136.001 GiB 667813 / 667791 items 0 errors ETA 0:00 duration: 1:48:16, 21.44MiB/s snapshot 9abedefd saved real 108m23.314s user 48m2.328s sys 6m12.984s * Second pass Shell# time ./restic_0.7.1_linux_amd64 -r /path/to/BackupTests/Restic/ backup /path/to/Mail/ enter password for repository: using parent snapshot 9abedefd scan [/path/to/Mail] scanned 2327 directories, 665575 files in 0:04 [0:47] 100.00% 2.855 GiB/s 136.010 GiB / 136.010 GiB 667902 / 667902 items 0 errors ETA 0:00 duration: 0:47, 2920.94MiB/s snapshot 6c90edf6 saved real 0m55.859s user 2m10.312s sys 0m9.364s BORG vs RESTIC on Backup : ========================== - Borg is way faster on first pass (1h15m vs 1h48m) but significantly slower on second pass (2m1s vs 47s) - Borg repo size (103 GB) is smaller than Restic repo size (121 GB) BORG vs RESTIC on mounted archives : ==================================== * Simple access to the mounted repositories : Shell# time ls -l BorgMount/20170911-181622/ total 0 drwxr-xr-x 1 root root 0 sept. 11 19:48 path real 0m22.383s user 0m0.000s sys 0m0.000s Shell# time ls -l ResticMount/snapshots/2017-09-11T18\:15\:18+02\:00/ total 0 drwx------ 3 mail mail 0 d?c. 7 2014 Mail real 0m0.003s user 0m0.000s sys 0m0.000s - Borg needs 22 seconds to internally build the directory tree, Restic is instant. - Interesting note : The first visible directory is exactly the specified backup path "path" (/path/to/Mail) for Borg whereas Restic only keeps the last path component "Mail" (/path/to/Mail). * Extract some path from the mounted repositories : - Shell# time cp -a BorgMount/20170911-181622/path/to/[...]/Trash BorgRestore/ real 3m36.534s user 0m0.396s sys 0m7.944s NOTE: CPU usage was spiking at 100% when no disk activity (building internal listings is my guess) and jumping between 31~67% for disk activity (actual copy process) - Shell# time cp -a ResticMount/snapshots/2017-09-11T18\:15\:18+02\:00/[...]/Trash ResticRestore/ real 6m23.970s user 0m0.496s sys 0m13.708s NOTE: CPU usage never spikes and constantly jumps between 21~53% for the whole process - The "Trash" directory is 6.3 GB big with 47945 files in it. - Borg is faster by a factor of 2 to restore the exact same data using about 2x more CPU. * Fetch deep info on mounted repositories : Shell# time du -s --si ResticMount/snapshots/2017-09-11T18\:15\:18+02\:00/ 147G ResticMount/snapshots/2017-09-11T18:15:18+02:00/ real 1m18.590s user 0m0.800s sys 0m4.036s NOTE: CPU usage around 46% for the whole process Shell# time du -s --si BorgMount/20170911-181622/ 138G BorgMount/20170911-181622/ real 5m30.143s user 0m0.864s sys 0m4.956s NOTE: CPU usage at 100% for the whole process - BORG is about 5x slower to get the same information BORG vs RESTIC trying to backup while having mounted archives : =============================================================== NOTE: Typical use case would be trying to restore a very big file in a very nested/complex directory hierarchy that would make this impractical using the "extract/restore" command. Retrieving the said file would be so time consuming that it would overlap with the next scheduled backup for example. Shell# ./borg-linux64 create --info --stats --progress /path/to/BackupTests/Borg::{now:%Y%m%d-%H%M%S} /path/to/Mail/ Failed to create/acquire the lock /path/to/BackupTests/Borg/lock (timeout). Shell# ./restic_0.7.1_linux_amd64 -r /path/to/BackupTests/Restic backup /path/to/Mail/ enter password for repository: using parent snapshot 6c90edf6 scan [/path/to/Mail] scanned 2327 directories, 665655 files in 0:03 [0:38] 100.00% 3.518 GiB/s 136.039 GiB / 136.039 GiB 667982 / 667982 items 0 errors ETA 0:00 duration: 0:38, 3581.52MiB/s snapshot 64106e49 saved NOTE: Here the Restic design have a clear advantage. Quoting the doc : "All files in a repository are only written once and never modified afterwards. This allows accessing and even writing to the repository with multiple clients in parallel". Features I like in Restic : =========================== - Nothing is written outside the repository - The "views" on mounted repositories (host, snapshots and tags) - Multiple "keys" per repository (like LUKS) Features I *DISLIKE* in Restic : ================================ - The "mount" command blocks the shell and waits for "CTRL-C" to end. Trying to unmount while mountpoint is busy (ie: cd /path/to/mountpoint in a different shell) ends up badly : "unable to umount (maybe already umounted?): exit status 1: fusermount: failed to unmount /path/to/BackupTests/ResticMount: Device or resource busy" and needs manual intervention. Borg also complains but a second invocation to "umount" command works once the "business" state is lifted. - No support for sparse files (AFAIK) which makes it not usable for VM images and such. Features I *DISLIKE* in Borg : ============================== - Writes several files OUTSIDE the repository, ~/.config/borg and ~/.cache/borg and AFAIK, there's no option to use another paths for these files. - The "several seconds or more" delays when mounting repositories and scanning deeper directories. I can live with the delays but I really wish there was an option to relocate the ".config" and ".cache" data. I need this because it makes it easier to copy the data offsite without forgetting anything! I know that ".cache" is disposable bug having this data available when restoring in case of disaster recovery is a huge gain of time. Features I *DISLIKE* in BOTH tools : ==================================== - Their design geared at "backup-and-push-to-repository" which is nice but not desired in my environment. I need a "repository-pulls-backup-from-agent" design. There could be in both tools an additional "agent" command that would : * Use ssh transport by default to contact an host and the ssh keys benefits (authorized keys, ) * Spawn a Borg/Restic instance to make the backup on the remote host (like a normal Borg call) but feed the result back to the calling Borg, which holds the repository * A way to securely transmit the repokey data to the remote instance so the local Borg can mount/check the local repository Of course, it would be of the administrator responsability to setup everything accordingly to use either one repokey for every remote host or script something a bit smarter to use a repokey per host or group of hosts, whatever suits the needs. Why such a setup? Because, in my case at least, the backup server is of critical importance and network isolated from the other hosts. I really don't want the "all-hosts-can-contact-the-backup-server" style but the "only-backup-server-can-contact-hosts" kind of behavior. This also helps to limit the strain on the backup server. Having all the hosts, with no predictable backup size, hammering the backup server at the same time (cronjob) is not desirable, especially on sites with storage on budget :-) For instance, I currently use a very spartan/crude system but which is rock solid and never failed once in over two decades. A simple script which, in sequence, connects via SSH to each host and uses the remote tar command to perform the backup. SSH's piped stdout/stderr allows to retrieve the tarball as well as errors and act accordingly. This is not scalable but highly effective, battle tested and disaster recovery proven! Booting a new server with some rescue OS and restoring from a tarball works in ALL conditions, no matter how long it takes :-) But now, I need encryption and deduplication given the huge sizes of the data to backup, hence my tests with Borg/Restic which both have nice features *AND* provide a single file binary for disaster scenarios. -- Unix _IS_ user friendly, it's just selective about who its friends are. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Mon Sep 11 23:37:46 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 12 Sep 2017 05:37:46 +0200 Subject: [Borgbackup] Test : Borg vs Restic In-Reply-To: References: Message-ID: <0a354364-4c99-611b-a3d1-cf0dfa7399c2@waldmann-edv.de> Hi Melkor, > The backup data consists of a live mail repository using the maildir > format and holding 139 GB (2327 dirs, 665456 files). == lots of little files in a huge directory, performance will be relatively bad in any case. > This archive: 137.47 GB 112.55 GB 102.30 GB Compression is relatively bad because it's only small files. > This archive: 137.47 GB 112.55 GB 14.57 MB Hmm, maybe your first run was not as root, thus it changed atime, thus not deduplicating the metadata stream. Maybe try running as root (then it can open using a no-atime-changing mode) or borg create --no-atime to not store the atime. > (restic) duration: 0:47, 2920.94MiB/s Hah, so much for meaningless performance metrics (it did not really READ the files, so the 2.9GiB/s didn't really happen). :D > - Borg is way faster on first pass (1h15m vs 1h48m) but significantly > slower on second pass (2m1s vs 47s) As you see at the deduped size, borg wrote some new information to the repo (likely changed atime timestamps). Not sure whether restic deals with changed atime / does store it at all. > - Borg repo size (103 GB) is smaller than Restic repo size (121 GB) Expected. Restic can't compress, borg 1.1 does lz4 by default. > BORG vs RESTIC on mounted archives : > ==================================== > borg real 0m22.383s > restic real 0m0.003s Expected. A borg archive is somehow like a tar archive (but only the metadata as one stream, having the content data stored separately). So it is one linear stream of file metadata for all the stuff. There is no separate directory information, directory contents are just implicit from all the file paths. When doing the FUSE mount, borg needs to read the whole archive and build up the hierarchical filesystem structure in memory. Restic stores directory information in a separate / structured way, so I guess it loads just what it needs on demand when you enter a new directory. Maybe run a ls -lR on both mounts and include that in the timing. > * Extract some path from the mounted repositories : > borg real 3m36.534s > restic real 6m23.970s The revenge of the lazy loading? :D > - Borg is faster by a factor of 2 to restore the exact same data using > about 2x more CPU. BTW, when extracting bigger amounts, borg extract should be faster than copying from the fuse mount. > * Fetch deep info on mounted repositories : > Shell# time du -s --si ... > restic real 1m18.590s > borg real 5m30.143s Hmm, that is strange. For old borg (< 1.1) I'ld have said "ok, it needs to compute size information by summing up chunk lengths". But IIRC we now store that information into the metadata (1.1), so that should be fast now. But maybe it is something completely different. > BORG vs RESTIC trying to backup while having mounted archives : We play safe. Whenever you do something, the repo is locked. This is a quite coarse granular way, but simple and safe. For multiple writers, a lock is rather obviously needed to avoid corrupting shared data structures. But even for a one writer multiple reader scenario it is unsafe to read data while a writer could modify it... > NOTE: Typical use case would be trying to restore a very big file in a > very nested/complex directory hierarchy that would make this impractical > using the "extract/restore" command. Retrieving the said file would be > so time consuming that it would overlap with the next scheduled backup > for example. We might be able to do less / more fine granular locking in the future, but next priority is multihreading to increase performance and crypto enhancements, so locking will have to wait. > NOTE: Here the Restic design have a clear advantage. Quoting the doc : > "All files in a repository are only written once and never modified > afterwards. This allows accessing and even writing to the repository > with multiple clients in parallel". For all the content data chunks and backup archives, this is quite the same for borg. The manifest (list of all archives) is read-modify-write though. Also, when writing with multiple writers at the same time, there is obviously a cache coherency issue (if one uses a local cache, like borg does). Not necessarily fatal, but maybe less efficient. > Features I like in Restic : > =========================== > > - Nothing is written outside the repository If you mean borg's cache with "outside the repository": this is for performance reasons. We could always query the repo, but that would be slower. > - The "views" on mounted repositories (host, snapshots and tags) Yes, borg just has archive names and comments. Maybe a little less versatile / structured, but does its job (assuming you use sane names). > - Multiple "keys" per repository (like LUKS) So it is the same repo encryption key, just encrypted with different passphrases? > Features I *DISLIKE* in Restic : > ================================ > > - The "mount" command blocks the shell and waits for "CTRL-C" to end. Likely: Just not forking to the background. Good for debugging, bad for UX. > - No support for sparse files (AFAIK) which makes it not usable for VM > images and such. Oh, really? I'ld have expected that since it is easy to do (at least the simple implementation). > Features I *DISLIKE* in Borg : > ============================== > > - Writes several files OUTSIDE the repository, ~/.config/borg and > ~/.cache/borg and AFAIK, there's no option to use another paths for > these files. There is, see environment vars. > - The "several seconds or more" delays when mounting repositories and > scanning deeper directories. Lazy loading for all directory levels is not efficiently possible for borg due to the linear archive structure. > I can live with the delays but I really wish there was an option to > relocate the ".config" and ".cache" data. I need this because it makes > it easier to copy the data offsite without forgetting anything! You don't need to copy the cache, it can be reconstructed (just takes some time). But make sure you do not lose passphrase and key / make a backup of them (borg key ...). > I know > that ".cache" is disposable bug having this data available when > restoring in case of disaster recovery is a huge gain of time. Hmm, did you actually try that? > Features I *DISLIKE* in BOTH tools : > ==================================== > > - Their design geared at "backup-and-push-to-repository" which is nice > but not desired in my environment. I need a > "repository-pulls-backup-from-agent" design. Matter of taste / threat model. We have an issue in the issue tracker about it with some suggestion how to implement pull (based on existing code). It just needs somebody to confirm it and do a docs pull request. Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From fabio.pedretti at unibs.it Tue Sep 12 04:11:23 2017 From: fabio.pedretti at unibs.it (Fabio Pedretti) Date: Tue, 12 Sep 2017 10:11:23 +0200 Subject: [Borgbackup] Test : Borg vs Restic In-Reply-To: References: Message-ID: [...] 2017-09-12 4:12 GMT+02:00 Melkor Lord : > Features I *DISLIKE* in BOTH tools : > ==================================== > > - Their design geared at "backup-and-push-to-repository" which is nice but > not desired in my environment. I need a "repository-pulls-backup-from-agent" > design. There could be in both tools an additional "agent" command that > would : > * Use ssh transport by default to contact an host and the ssh keys > benefits (authorized keys, ) > * Spawn a Borg/Restic instance to make the backup on the remote host (like > a normal Borg call) but feed the result back to the calling Borg, which > holds the repository > * A way to securely transmit the repokey data to the remote instance so > the local Borg can mount/check the local repository > > Of course, it would be of the administrator responsability to setup > everything accordingly to use either one repokey for every remote host or > script something a bit smarter to use a repokey per host or group of hosts, > whatever suits the needs. > > Why such a setup? > > Because, in my case at least, the backup server is of critical importance > and network isolated from the other hosts. I really don't want the > "all-hosts-can-contact-the-backup-server" style but the > "only-backup-server-can-contact-hosts" kind of behavior. This also helps to > limit the strain on the backup server. Having all the hosts, with no > predictable backup size, hammering the backup server at the same time > (cronjob) is not desirable, especially on sites with storage on budget :-) > > For instance, I currently use a very spartan/crude system but which is > rock solid and never failed once in over two decades. A simple script which, > in sequence, connects via SSH to each host and uses the remote tar command > to perform the backup. SSH's piped stdout/stderr allows to retrieve the > tarball as well as errors and act accordingly. This is not scalable but > highly effective, battle tested and disaster recovery proven! Booting a new > server with some rescue OS and restoring from a tarball works in ALL > conditions, no matter how long it takes :-) But now, I need encryption and > deduplication given the huge sizes of the data to backup, hence my tests > with Borg/Restic which both have nice features *AND* provide a single file > binary for disaster scenarios. I also had that requirement and can be done with borg. I did set up a borg server mounting via nfs/sshfs the hosts to backup, then you can schedule all the backups in a sequential way from the borg server with crontab. Having only a single concurrent backup you can also have a shared repository for all hosts to backup, potentially increasing deduplication. Also no need to have a borg client on the hosts to backup, just ssh for sshfs or nfs server (another benefit is that you have to update only one borg binary when a new release is out). -- ing. Fabio Pedretti Responsabile U.O.C. "Reti e Sistemi" http://www.unibs.it/organizzazione/amministrazione-centrale/servizio-servizi-ict/uoc-reti-e-sistemi Universit? degli Studi di Brescia Via Valotti, 9 - 25121 Brescia E-mail: fabio.pedretti at unibs.it -- Informativa sulla Privacy: http://www.unibs.it/node/8155 From maurice.libes at osupytheas.fr Tue Sep 12 04:12:49 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Tue, 12 Sep 2017 10:12:49 +0200 Subject: [Borgbackup] question about deduplicated size stats Message-ID: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> hi to all just a question yesterday I made a backup here below, the deduplicated size was 103.49G Archive name:/*baklouti-homes-2017-09-11*/ Time (start): Mon, 2017-09-11 12:27:09 Time (end): Mon, 2017-09-11 16:41:53 Duration: 4 hours 14 minutes 44.46 seconds Number of files: 768993 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive:/* 347.38 GB 139.32 GB 103.49 GB*/ All archives: 347.38 GB 139.32 GB 103.49 GB Unique chunks Total chunks Chunk index: 695181 864806 ------------------------------------------------------------------------------ in the night there have been another backup /*baklouti-homes-2017-09-12*/ but if I want to display again the stats of the same backup of yesterday */baklouti-homes-2017-09-11/* with a borg info command, now it displays /*39.20MB*/ why the size have changed ? thanks ML * * *$ bo**/rg info borg at myserver.fr:/mnt/provigo-borg/sauve-pcbaklouti::baklouti-homes-2017-09-11/* Name: baklouti-homes-2017-09-11 Hostname: baklouti-Precision-7810 Username: root Time (start): Mon, 2017-09-11 12:27:09 Time (end): Mon, 2017-09-11 16:41:53 Command line: /usr/local/bin/borg create -v --stats --compression zlib,3 borg at myserver.fr:/mnt/provigo-borg/sauve-pcbaklouti::/*baklouti-homes-2017-09-11*/ /home/baklouti/ --exclude /home/*/.cache Number of files: 768993 Original size Compressed size Deduplicated size This archive:/* 347.38 GB 139.32 GB 39.20 MB*/ All archives: 694.77 GB 278.64 GB 103.53 GB Unique chunks Total chunks Chunk index: 695314 1729578 -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From griffiths_andy at icloud.com Tue Sep 12 05:52:59 2017 From: griffiths_andy at icloud.com (Andrew Griffiths) Date: Tue, 12 Sep 2017 10:52:59 +0100 Subject: [Borgbackup] How to recover from incorrect index file length error? Message-ID: <97B1ECD8-7FE1-4942-AF4A-01AA01EE5C92@icloud.com> My repo volume ran out of space last night, and now Borg reports an invalid index file length. This has happened to me before, and on that occasion I tried running borg check (for over 24 hours, 1TB over network) but did not fix the issue. I had to start the repo from scratch. Is there any way of fixing this? andyg at Andys-MBA ~ $ bin/borg list /Volumes/andyg/borg/ hashindex: /Volumes/andyg/borg/index.144258: Incorrect file length (expected 87578378, got 84017152) Local Exception. Traceback (most recent call last): File "borg/archiver.py", line 2168, in main File "borg/archiver.py", line 2104, in run File "borg/archiver.py", line 101, in wrapper File "borg/helpers.py", line 247, in load File "borg/repository.py", line 522, in get File "borg/repository.py", line 214, in open_index File "borg/hashindex.pyx", line 79, in borg.hashindex.IndexBase.read File "borg/hashindex.pyx", line 67, in borg.hashindex.IndexBase.__cinit__ Exception: hashindex_read failed Platform: Darwin Andys-MBA 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64 i386 Borg: 1.0.11 Python: CPython 3.5.3 PID: 46808 CWD: /Users/andyg sys.argv: ['bin/borg', 'list', '/Volumes/andyg/borg/'] SSH_ORIGINAL_COMMAND: None andyg at Andys-MBA ~ $ ls -l /Volumes/andyg/borg/ total 165856 -rwx------ 1 andyg staff 26 5 Sep 17:40 README -rwx------ 1 andyg staff 164 5 Sep 17:40 config drwx------ 1 andyg staff 16384 11 Sep 03:17 data -rwx------ 1 andyg staff 719664 12 Sep 01:43 hints.144258 -rwx------ 1 andyg staff 84017152 12 Sep 01:43 index.144258 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Tue Sep 12 07:52:41 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 12 Sep 2017 13:52:41 +0200 Subject: [Borgbackup] How to recover from incorrect index file length error? In-Reply-To: <97B1ECD8-7FE1-4942-AF4A-01AA01EE5C92@icloud.com> References: <97B1ECD8-7FE1-4942-AF4A-01AA01EE5C92@icloud.com> Message-ID: <53ef9718-3fc6-3cc4-34c2-accec099ad1d@waldmann-edv.de> On 09/12/2017 11:52 AM, Andrew Griffiths wrote: > My repo volume ran out of space last night, You should try to avoid that (see docs). > and now Borg reports an invalid index file length. You should make quite some space free otherwise, delete index.144258 and hints.144258 and then run borg check --repair (it will rebuild the index and that will take a while). borg always needs some free space to be able to work - even to delete archives. some commands have a --save-space option that makes them slower, but less space-consuming. borg 1.1 will have some new code that tries to deal better with (close to) out-of-space conditions. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From tw at waldmann-edv.de Tue Sep 12 08:00:54 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 12 Sep 2017 14:00:54 +0200 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> Message-ID: <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> > why the size have changed ? Because the deduplicated size (== size of all chunks ONLY used by this 1 archive) is relative to the state of the whole repository. So if you put these files into these archives (nothing else in that repo, A, B, C have no duplicate chunks): A -> archive1 A B -> archive2 A B C -> archive3 After creating archive2, the new chunks from B will show up as unique chunks of archive2 (because they are not referenced by archive1). But after you create archive3, the chunks from B will be referenced twice (by archive2 and 3), thus they are not uniquely referenced by archive2 any more. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From griffiths_andy at icloud.com Tue Sep 12 11:41:35 2017 From: griffiths_andy at icloud.com (Andrew Griffiths) Date: Tue, 12 Sep 2017 16:41:35 +0100 Subject: [Borgbackup] How to recover from incorrect index file length error? Message-ID: <1A99C5EC-5511-4061-A8B6-9C04B4C8F2F0@icloud.com> Thanks Thomas. Trying that now. From maurice.libes at osupytheas.fr Wed Sep 13 07:10:40 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Wed, 13 Sep 2017 13:10:40 +0200 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> Message-ID: Le 12/09/2017 ? 14:00, Thomas Waldmann a ?crit : >> why the size have changed ? > Because the deduplicated size (== size of all chunks ONLY used by this 1 > archive) is relative to the state of the whole repository. > > So if you put these files into these archives (nothing else in that > repo, A, B, C have no duplicate chunks): > > A -> archive1 > A B -> archive2 > A B C -> archive3 > > After creating archive2, the new chunks from B will show up as unique > chunks of archive2 (because they are not referenced by archive1). > > But after you create archive3, the chunks from B will be referenced > twice (by archive2 and 3), thus they are not uniquely referenced by > archive2 any more. > > ok I have "almost" understood, so the size returned by /*"borg info"*/ displays the stats of the deduplicated size (unique fragments) of one archive as compared to all the fragments in the whole repository ? so it's a relative proportion of the unique fragments of one archive , which can evolve in the time when new archives are added in the repository ? After a great reflexion :-) this is very confusing for my mind (and some friends) because we expected to find the deduplicated size information as returned by the/*borg create*/ command at the time the backup was made that is to say the deduplicated size of one archive at the time it is made which is to my opinion more interesting to see the benefit of deduplication exemple time of "/_borg create_/" (extracted from log file) Archive name: bioinfo-2017-09-07/** Original size Compressed size Deduplicated size This archive: 6.44 TB 4.02 TB *2.44 GB* All archives: 65.05 TB 41.60 TB 3.96 TB**/ time of "_/borg info/_" some weeks later Name: bioinfo-2017-09-07 Original size Compressed size Deduplicated size This archive: 6.44 TB 4.02 TB/*110.13 MB*/ All archives: 66.26 TB 41.89 TB 4.29 TB some weeks later we lose this information which shows that the day of the archive (09-07), borg added 2.44Gb of new fragments in the repository we only know that in the 09-07 archive there are 110.13 Mb of unique fragments from this archive in the whole repository dont know if i am clear (difficult to conceptualize and translate in english) your opinion ? ML -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From gmatht at gmail.com Wed Sep 13 09:08:39 2017 From: gmatht at gmail.com (John McCabe-Dansted) Date: Wed, 13 Sep 2017 21:08:39 +0800 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> Message-ID: > > After a great reflexion :-) this is very confusing for my mind (and some > friends) because we expected to find > the deduplicated size information as returned by the* borg create* > command at the time the backup was made > > that is to say the deduplicated size of one archive at the time it is made > which is to my opinion more interesting to see the benefit of deduplication > It can be confusing that they differ, but as it stands it reflects how much space would be freed if you deleted that one archive. I think that the easiest way if you want the the original value of the "Deduplicated size" is to log the output of borgbackup. I guess the other statistic that would be of interest is how much space would be freed if some set of archives were deleted. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisdowning250 at gmail.com Wed Sep 13 09:26:04 2017 From: chrisdowning250 at gmail.com (Chris Downing) Date: Wed, 13 Sep 2017 13:26:04 +0000 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> Message-ID: Just jumping in here - not much experience with Borg but I think I understand the query... It sounds like what Maurice is after is a way to answer: "If all archives except this one were not present, what would be the deduplicated size of this archive?" If correct, I agree that it would be a useful data point, but have no idea how complex it would be to generate that information (assuming it doesn't already exist somewhere). On Wed, 13 Sep 2017 at 14:09 John McCabe-Dansted wrote: > After a great reflexion :-) this is very confusing for my mind (and some >> friends) because we expected to find >> the deduplicated size information as returned by the* borg create* >> command at the time the backup was made >> >> that is to say the deduplicated size of one archive at the time it is made >> which is to my opinion more interesting to see the benefit of >> deduplication >> > > > It can be confusing that they differ, but as it stands it reflects how > much space would be freed if you deleted that one archive. > > I think that the easiest way if you want the the original value of the > "Deduplicated size" is to log the output of borgbackup. > > I guess the other statistic that would be of interest is how much space > would be freed if some set of archives were deleted. > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Wed Sep 13 09:46:49 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 13 Sep 2017 15:46:49 +0200 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> Message-ID: <57772724-6843-f5db-fec4-8e4ac22892c8@waldmann-edv.de> > It sounds like what Maurice is after is a way to answer: "If all > archives except this one were not present, what would be the > deduplicated size of this archive?" If there is no other archive, there is only inner deduplication, but no historical inter-archive dedup. But borg does not compute a separate value for (only) the inner deduplication inside an archive. > If correct, I agree that it would be a useful data point, but have no > idea how complex it would be to generate that information (assuming it > doesn't already exist somewhere). It could be computed by reading this archive's metadata and creating a chunks index just from that. Not sure if it would not even confuse people more than they already are when it comes to dedup statistics. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From public at enkore.de Wed Sep 13 10:02:01 2017 From: public at enkore.de (Marian Beermann) Date: Wed, 13 Sep 2017 16:02:01 +0200 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: <57772724-6843-f5db-fec4-8e4ac22892c8@waldmann-edv.de> References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> <57772724-6843-f5db-fec4-8e4ac22892c8@waldmann-edv.de> Message-ID: <00fff680-0155-2e6d-cc45-16ef37ea73ae@enkore.de> On 13.09.2017 15:46, Thomas Waldmann wrote: >> It sounds like what Maurice is after is a way to answer: "If all >> archives except this one were not present, what would be the >> deduplicated size of this archive?" > > If there is no other archive, there is only inner deduplication, but no > historical inter-archive dedup. > > But borg does not compute a separate value for (only) the inner > deduplication inside an archive. This value would still differ from what "borg create" had shown. I suppose it could store that number in the archive metadata, but it of course breaks down with recreate etc. From maurice.libes at osupytheas.fr Wed Sep 13 11:21:32 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Wed, 13 Sep 2017 17:21:32 +0200 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> Message-ID: <4ad3edc4-c7c7-b15f-a110-469d0d71b158@osupytheas.fr> Le 13/09/2017 ? 15:26, Chris Downing a ?crit : > Just jumping in here - not much experience with Borg but I think I > understand the query... > > It sounds like what Maurice is after is a way to answer: "If all > archives except this one were not present, what would be the > deduplicated size of this archive?" sorry, I dont know if this was I wanted to say :-) I just wanted to see somewhere in the stats (elsewhere than in the log file) the informations of the deduplicated size that "/borg create"/ displays after an archive backup; i.e the volume of new fragments found at time "t" for one particular archive at the moment where it is done but I am sensible to this argument of john: />It can be confusing that they differ, but as it stands it reflects how much space would be freed if you deleted >that one archive. /what is confusing for me is that , it is the same name deduplicated size, but not computed in the same manner between "borg create" and "borg info/" / / / let's say If I want to show the benefit of borg deduplication day after day , it would of some interest to display somewhere the B dedup volume for archive2 and the C dedup volume for archive3 A -> archive1 A B -> archive2 A B C -> archive3 dont know if it is a point of interest, It's up to you maybe explain this somewhere in the documentation? or in the man file (now I know I have to check what I search in the log file) thanks for the thread M// / ========= /time of "/_borg create_/" (extracted from log file) Archive name: bioinfo-2017-09-07/** Original size Compressed size Deduplicated size This archive: 6.44 TB 4.02 TB *2.44 GB* All archives: 65.05 TB 41.60 TB 3.96 TB**/ time of "_/borg info/_" some weeks later Name: bioinfo-2017-09-07 Original size Compressed size Deduplicated size This archive: 6.44 TB 4.02 TB/*110.13 MB*/ All archives: 66.26 TB 41.89 TB 4.29 TB > > If correct, I agree that it would be a useful data point, but have no > idea how complex it would be to generate that information (assuming it > doesn't already exist somewhere). > > On Wed, 13 Sep 2017 at 14:09 John McCabe-Dansted > wrote: > > After a great reflexion :-) this is very confusing for my > mind (and some friends) because we expected to find > the deduplicated size information as returned by the/*borg > create*/ command at the time the backup was made > > that is to say the deduplicated size of one archive at the > time it is made > which is to my opinion more interesting to see the benefit of > deduplication > > > > It can be confusing that they differ, but as it stands it reflects > how much space would be freed if you deleted that one archive. > > I think that the easiest way if you want the the original value of > the "Deduplicated size" is to log the output of borgbackup. > > I guess the other statistic that would be of interest is how much > space would be freed if some set of archives were deleted. > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From public at enkore.de Wed Sep 13 15:26:03 2017 From: public at enkore.de (Marian Beermann) Date: Wed, 13 Sep 2017 21:26:03 +0200 Subject: [Borgbackup] question about deduplicated size stats In-Reply-To: <4ad3edc4-c7c7-b15f-a110-469d0d71b158@osupytheas.fr> References: <27147092-e5a5-6c7d-0498-aef0a8378abd@osupytheas.fr> <493ebc71-8b04-3e17-086a-cd08203e01ac@waldmann-edv.de> <4ad3edc4-c7c7-b15f-a110-469d0d71b158@osupytheas.fr> Message-ID: <8bcf3b7c-f2ae-fd40-d16d-e8453cd37ea1@enkore.de> On 13.09.2017 17:21, Maurice Libes wrote: > > > Le 13/09/2017 ? 15:26, Chris Downing a ?crit : >> Just jumping in here - not much experience with Borg but I think I >> understand the query... >> >> It sounds like what Maurice is after is a way to answer: "If all >> archives except this one were not present, what would be the >> deduplicated size of this archive?" > sorry, I dont know if this was I wanted to say :-) > > I just wanted to see somewhere in the stats (elsewhere than in the log > file) the informations of the > deduplicated size that "/borg create"/ displays after an archive backup; > > i.e the volume of new fragments found at time "t" for one particular > archive at the moment where it is done > > > but I am sensible to this argument of john: > />It can be confusing that they differ, but as it stands it reflects how > much space would be freed if you deleted >that one archive. > > /what is confusing for me is that , it is the same name deduplicated > size, but not computed in the same manner > between "borg create" and "borg info/" > It is computed in the same manner, but at different states of the system, yielding different results. From dave at artlauncher.online Wed Sep 13 20:34:18 2017 From: dave at artlauncher.online (David Thibault) Date: Wed, 13 Sep 2017 17:34:18 -0700 Subject: [Borgbackup] Question about borg remote/local setup In-Reply-To: References: <88255ba0-4a84-d63e-c1b3-edf4b11aa2b6@waldmann-edv.de> Message-ID: I thought I'd follow up on this. I now have a solution where it's working very quickly on successive backups after the first one. Here's my setup: /usr/local/borg --borg repository on local hard disk /mnt/efs_borg --NFS (actually Amazon EFS) share with Borg data folder on it. /mnt/efs_prod --NFS share with production source data to be backed up. symlink /usr/local/borg/data to /mnt/efs_borg/data. That way all data is on the EFS share and the other index-related stuff is on the local drive. Backing up /mnt/efs_prod initially took about 45 minutes for 2.5 GB. Second backup took under two minutes. I back up /mnt/efs_prod and then I back up everything in /usr/local/borg and exclude /usr/local/borg/data. Then, for easy restores, I tar everything in /usr/local/borg except data and copy that tar file to /mnt/efs_borg/. This last tar file just overwrites the last one, but all I would need it for would be to This seems to work great. Any thoughts, concerns, etc, about using it like this with the index files local and the data folder remote? Any concerns for disaster recovery like this? What happens if I lose everything on the local drive? Can I rebuild an index from just the data folder on NFS? Thanks, Dave On Tue, Aug 22, 2017 at 10:53 AM, David Thibault wrote: > Thanks for the quick response! > > > On Tue, Aug 22, 2017 at 10:37 AM, Thomas Waldmann > wrote: > >> > I've got Borg working with all of the files (data, indexes, and >> > everything) on an NFS share (actually, it's Amazon EFS). It's actually >> > backing up image data from one NFS share (my WP Uploads files) to the >> > borg repo on a different NFS share. >> > >> > It works fine, but it takes about 45 minutes to run through less than >> > 3GB of image data, even on subsequent runs. >> >> That's likely because it reads/chunks everything because it thinks the >> file has changed. This is when one of mtime, inode nr, size is not >> stable (usually the inode nr is not stable on network filesystems). >> >> If you can trust only mtime and size for the "is the file changed" >> detection, you can tell borg to ignore the inode number (see >> --ignore-inode or so). >> >> > I'm guessing the process >> > would be much faster if I stored the Borg metadata on the local server >> > and only the data blocks on NFS. Then, after the main borg backup, I >> > could do a cron script that would zip up the remaining borg metadata >> > files, then borg could back that zip file as well. >> >> Not sure how you mean that, but in general local storage is usually >> faster than network storage. >> >> Also have a look at cache / index sizes and maybe increase borg 1.0.x >> --checkpoint-interval from 300 (5mins) to 1800 (30mins) to have fewer >> checkpoints but also less overhead - esp. if indexes/caches are large >> and reading/writing them is slow. >> >> > If you agree, which files would be best kept on the local disk and which >> > should be on NFS. Judging by the repo contents, I'm guessing the data >> > folder on NFS and the rest on the local disk. Can you confirm? >> >> Maybe try the --checkpoint-interval first, then you maybe do not have to >> do that. >> >> >> -- >> >> GPG ID: 9F88FB52FAF7B393 >> GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Thu Sep 14 14:44:49 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 14 Sep 2017 20:44:49 +0200 Subject: [Borgbackup] Question about borg remote/local setup In-Reply-To: References: <88255ba0-4a84-d63e-c1b3-edf4b11aa2b6@waldmann-edv.de> Message-ID: <2aae51ca-5d7c-1111-b8a7-4527670626ea@waldmann-edv.de> > symlink /usr/local/borg/data to /mnt/efs_borg/data. That way all data > is on the EFS share and the other index-related stuff is on the local drive. Please note that for "repokey" encryption mode (default), the file containing the key (/config) is, in your case, on your local disk. For recovery, you must have the passphrase and the key, so be careful to have a backup (see borg key ...). > I back up /mnt/efs_prod and then I back up everything in /usr/local/borg > and exclude /usr/local/borg/data. Then, for easy restores, I tar > everything in /usr/local/borg except data and copy that tar file to > /mnt/efs_borg/. Ah, ok, then you have the config (key) in there. > This seems to work great. Any thoughts, concerns, etc, about using it > like this with the index files local and the data folder remote? As long as the filesystems work as they should and you can always recover data, key and passphrase, this seems ok. > Can I rebuild an index from just the data folder on NFS? All caches and index files can be rebuilt from repo/data. It may take a while though, depending on repo size. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From dave at artlauncher.online Thu Sep 14 17:53:53 2017 From: dave at artlauncher.online (David Thibault) Date: Thu, 14 Sep 2017 14:53:53 -0700 Subject: [Borgbackup] Question about borg remote/local setup In-Reply-To: <2aae51ca-5d7c-1111-b8a7-4527670626ea@waldmann-edv.de> References: <88255ba0-4a84-d63e-c1b3-edf4b11aa2b6@waldmann-edv.de> <2aae51ca-5d7c-1111-b8a7-4527670626ea@waldmann-edv.de> Message-ID: Actually, when I started this thread I was looking into using S3 instead of EFS, because it's way cheaper. S3 didn't work because it wanted to edit files. Then I switched to EFS. However, with this hybrid setup, keeping everything but the ./data folder on the local disk and the ./data folder on S3 does actually work...at least so far...it's creating block files in ./data. That's further than it got before. I'm thinking the hybrid approach with s3 mounted via RIOFS might be the way to go. Best, Dave On Thu, Sep 14, 2017 at 11:44 AM, Thomas Waldmann wrote: > > symlink /usr/local/borg/data to /mnt/efs_borg/data. That way all data > > is on the EFS share and the other index-related stuff is on the local > drive. > > Please note that for "repokey" encryption mode (default), the file > containing the key (/config) is, in your case, on your local disk. > > For recovery, you must have the passphrase and the key, so be careful to > have a backup (see borg key ...). > > > I back up /mnt/efs_prod and then I back up everything in /usr/local/borg > > and exclude /usr/local/borg/data. Then, for easy restores, I tar > > everything in /usr/local/borg except data and copy that tar file to > > /mnt/efs_borg/. > > Ah, ok, then you have the config (key) in there. > > > This seems to work great. Any thoughts, concerns, etc, about using it > > like this with the index files local and the data folder remote? > > As long as the filesystems work as they should and you can always > recover data, key and passphrase, this seems ok. > > > Can I rebuild an index from just the data folder on NFS? > > All caches and index files can be rebuilt from repo/data. > It may take a while though, depending on repo size. > > -- > > GPG ID: 9F88FB52FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > -------------- next part -------------- An HTML attachment was scrubbed... URL: From anarcat at debian.org Sat Sep 16 10:41:46 2017 From: anarcat at debian.org (=?utf-8?Q?Antoine_Beaupr=C3=A9?=) Date: Sat, 16 Sep 2017 10:41:46 -0400 Subject: [Borgbackup] variable purge policies per filesystem tree Message-ID: <87r2v61z9x.fsf@curie.anarc.at> Hi, I suspect this is impossible because of the way borg is designed, but is there a way to have different prune policies for different parts of the filesystem? For example, I would need to purge /var/log faster than the rest of the filesystem. I would rather drop those after a week, but I would keep other parts of the filesystem indefinitely. The only way I can think of doing this is having two separate set of archives within the repositories, with each its own purge policies. That feels rather clunky because I need to rely on some hardcoded special naming pattern on the archive names to purge (or not) certain archievs. Right now my naming scheme is, more or less: $(hostname)-$(date +%Y-%m-%d)$tag where $tag is a manual tag I could append to make more than one manual snapshot per day. Now, I am changing this to: $(hostname)-$tag-$(date +%Y-%m-%d) With $tag defaulting to "auto" in the script: tag=${1:-auto} It looks like a trivial change, but it allows me to use different purge policies thanks to the -P/--prefix parameter in borg purge. I also had to rename all archives to add the "auto" parameter: host=$(hostname) repo="." borg list "$repo" | sed "s/\($host-....-..-..\).*/\1/" | while read archive; do new=$(echo $archive | sed 's/$host-/$host-auto-/') echo borg rename "$repo"::"$archive" "$new" done (add "| sh" to that pipeline once you're sure you like the generated script) Then I can do the following purge policies: borg prune --verbose --list -d 30 -w 52 -y 10 -P $(hostname)-auto- "$repo" borg prune --verbose --list -d 7 -P $(hostname)-logs- "$repo" This is in my updated cron job available here: https://github.com/anarcat/community/tree/cron.daily Now the only question that remains is - how do i purge the logs from those automated archives... Thanks, A. -- Le travail des pauvres est la seule source de toutes les douceurs de l'existence. - Bernard Mandeville From tw at waldmann-edv.de Sat Sep 16 12:13:22 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 16 Sep 2017 18:13:22 +0200 Subject: [Borgbackup] variable purge policies per filesystem tree In-Reply-To: <87r2v61z9x.fsf@curie.anarc.at> References: <87r2v61z9x.fsf@curie.anarc.at> Message-ID: <134ad221-c1d1-3736-d6cf-260067ae3372@waldmann-edv.de> > I suspect this is impossible because of the way borg is designed, but is > there a way to have different prune policies for different parts of the > filesystem? No, not possible. It either deletes the whole archive or not, based on its timestamp and your pruning policy. > For example, I would need to purge /var/log faster than the rest of the > filesystem. I would rather drop those after a week, but I would keep > other parts of the filesystem indefinitely. I've seen that in your cron.daily script and I think the way you do it there (separate archives for logs and other stuff) is as good as it gets right now. > Now the only question that remains is - how do i purge the logs from > those automated archives... The only way to modify an existing archive will be "borg recreate" with different exclude param in borg 1.1. -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From public at enkore.de Sat Sep 16 12:35:43 2017 From: public at enkore.de (Marian Beermann) Date: Sat, 16 Sep 2017 18:35:43 +0200 Subject: [Borgbackup] variable purge policies per filesystem tree In-Reply-To: <87r2v61z9x.fsf@curie.anarc.at> References: <87r2v61z9x.fsf@curie.anarc.at> Message-ID: Hi Antoine, On 16.09.2017 16:41, Antoine Beaupr? wrote: > The only way I can think of doing this is having two separate set of > archives within the repositories, with each its own purge policies. > That feels rather clunky because I need to rely on some hardcoded > special naming pattern on the archive names to purge (or not) certain > archievs. > > Right now my naming scheme is, more or less: > > $(hostname)-$(date +%Y-%m-%d)$tag > > where $tag is a manual tag I could append to make more than one manual > snapshot per day. Now, I am changing this to: > > $(hostname)-$tag-$(date +%Y-%m-%d) > > With $tag defaulting to "auto" in the script: > > tag=${1:-auto} > > It looks like a trivial change, but it allows me to use different purge > policies thanks to the -P/--prefix parameter in borg purge. I also had > to rename all archives to add the "auto" parameter: I added --glob-archives some time ago to allow for more flexible ^W ^W sensible naming schemes, but it's only available in 1.1. From anarcat at debian.org Sun Sep 17 09:21:52 2017 From: anarcat at debian.org (Antoine =?utf-8?Q?Beaupr=C3=A9?=) Date: Sun, 17 Sep 2017 09:21:52 -0400 Subject: [Borgbackup] some performance numbers... Message-ID: <87a81t1mvj.fsf@curie.anarc.at> Yesterday I did my first purge ever of my repository. I had been doing daily backups of my home server for almost 3 years now and had 945 archives. It was an interesting experiment and I figured I would share the results I found. I ran the following command: time borg prune --verbose --list -d 30 -w 52 -y 10 $repo That deleted 860 archives and kept 85. It freed about 500GB of disk space and took well over 12 hours: real 798m5,477s user 341m9,688s sys 18m16,624s The last archive saved before the purge had the following stats: ------------------------------------------------------------------------------ Archive name: marcos-2017-09-15 Archive fingerprint: dbd7121462701d4916f3c4a6c9369154d75090d93be0f1a0c04a269b4902b435 Time (start): Fri, 2017-09-15 00:07:40 Time (end): Fri, 2017-09-15 00:28:10 Duration: 20 minutes 29.83 seconds Number of files: 1920931 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 231.90 GB 212.51 GB 1.11 GB All archives: 239.20 TB 211.46 TB 1.23 TB Unique chunks Total chunks Chunk index: 18349863 4937580530 ------------------------------------------------------------------------------ The next one had those: ------------------------------------------------------------------------------ Archive name: marcos-auto-2017-09-17 Archive fingerprint: 51637fc22df29c9e0dffe814de547e8c2c5045e3d41e0c318de5fbc37d442103 Time (start): Sun, 2017-09-17 00:09:45 Time (end): Sun, 2017-09-17 00:26:39 Duration: 16 minutes 54.47 seconds Number of files: 1919597 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive: 231.51 GB 212.15 GB 1.20 GB All archives: 19.31 TB 17.37 TB 550.97 GB Unique chunks Total chunks Chunk index: 8065589 397239183 ------------------------------------------------------------------------------ So an important takeaway here is that removing a significant portion of the archives improved the execution time by about 25% at almost zero cost in terms of information loss: notice how the number of files didn't really change, for example. This means we still track all files all the way back to the very beginning of the history here, and we can restore really old files. What we can't do is choose between two *days* for archives made 3 years ago, but the odds or needing that are dim. This also confirms we saved about 500GB of disk space during the purge, or, more accurately, about 650GB or 50% of the archive. Then I also had to rename archives to follow the new naming convention I use regarding logging (see previous mail). This also took a surprising amount of time... I had to rename the remaining 85 archives, and that took about an hour. So, there you go... Obviously, those are just some anectdotal results, but I figured it might be interesting to the casual readers here. A. -- We must shift America from a needs- to a desires-culture. People must be trained to desire, to want new things, even before the old have been entirely consumed. Man's desires must overshadow his needs. - Paul Mazur, Lehman Brothers From gait at atcomputing.nl Mon Sep 18 03:56:11 2017 From: gait at atcomputing.nl (Gerrit A. Smit) Date: Mon, 18 Sep 2017 09:56:11 +0200 Subject: [Borgbackup] Question about borg remote/local setup In-Reply-To: References: <88255ba0-4a84-d63e-c1b3-edf4b11aa2b6@waldmann-edv.de> <2aae51ca-5d7c-1111-b8a7-4527670626ea@waldmann-edv.de> Message-ID: David Thibault schreef op 14-09-17 om 23:53: > Actually, when I started this thread I was looking into using S3 > instead of EFS, because it's way cheaper. S3 didn't work because it > wanted to edit files. Then I switched to EFS. Hello David, Did you consider using rsync.net? That's an all ZFS operation with special tariff? when you use borg. http://rsync.net/products/attic.html Gerrit From dac at conceptual-analytics.com Mon Sep 18 14:59:20 2017 From: dac at conceptual-analytics.com (Dave Cottingham) Date: Mon, 18 Sep 2017 14:59:20 -0400 Subject: [Borgbackup] invalid ChunkerParams value Message-ID: Ran borg create with "--chunker-params 20,24,22,4095" and it says: borg create: error: argument --chunker-params: invalid ChunkerParams value: '20,24,22,4095' Why is this an invalid value? Thanks, Dave Cottingham -------------- next part -------------- An HTML attachment was scrubbed... URL: From maurice.libes at osupytheas.fr Tue Sep 19 04:11:23 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Tue, 19 Sep 2017 10:11:23 +0200 Subject: [Borgbackup] invalid ChunkerParams value In-Reply-To: References: Message-ID: <4a874fe3-b328-172c-1664-1ca45976493e@osupytheas.fr> Le 18/09/2017 ? 20:59, Dave Cottingham a ?crit : > Ran borg create with "--chunker-params 20,24,22,4095" and it says: > > borg create: error: argument --chunker-params: invalid ChunkerParams > value: '20,24,22,4095' > > Why is this an invalid value? strange, I made the same test yesterday, and it worked here is my command line borg create -v --stats/*--chunker-params 18,22,20,4095*/ \ borg at myserver.fr:/mnt/provigo-borg/testplus4Mchunk1M::sauve-09-18 /home/libes/Documents/testborg/ in my case I wanted to decrease the size of chunks to 1Mb (2^20) to see how much we could gain on a dataset of 137Gb (composed only by files greater than 4Mb) the results is that we gain just a little bit : 2Gb of dedup with chunks of 1Mb I imagined it could be more :-\ what is strange is that the duration time is lesser, but not very signiticative may be due to the server load? I would have thought that with a half chunk size , the duration time would be longer because there was the double of chunk here below my results I/ chunks of 2Mb default ================================== ------------------------------------------------------------------------------ Archive name: sauve-09-18 Time (start): Mon, 2017-09-18 12:25:25 Time (end): Mon, 2017-09-18 15:14:17 /*Duration: 2 hours 48 minutes 52.60 seconds*/ Number of files: 2395 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size This archive:/* 137.95 GB 137.95 GB 108.37 GB*/ All archives: 137.95 GB 137.95 GB 108.37 GB Unique chunks Total chunks Chunk index:/* 43245 55125*/ ------------------------------------------------------------------------------ II//*chunks of 1Mb 2^20*/ ================================== borg create -v --stats --chunker-params 18,22,20,4095 borg at myserver.fr: \ /mnt/provigo-borg/testplus4Mchunk1M::sauve-09-18 /home/libes/Documents/testborg/ ------------------------------------------------------------------------------ Archive name: sauve-09-18 Time (start): Mon, 2017-09-18 18:11:57 Time (end): Mon, 2017-09-18 20:56:40 /*Duration: 2 hours 44 minutes 42.61 seconds*/ Number of files: 2395 ------------------------------------------------------------------------------ Original size Compressed size Deduplicated size /*This archive: 137.95 GB 137.95 GB 106.94 GB*/ All archives: 137.95 GB 137.95 GB 106.94 GB Unique chunks Total chunks Chunk index:/* 83381 107820*/ ------------------------------------------------------------------------------ > > Thanks, > Dave Cottingham > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From steve at bstage.com Mon Sep 25 21:05:23 2017 From: steve at bstage.com (Steve Schow) Date: Mon, 25 Sep 2017 19:05:23 -0600 Subject: [Borgbackup] how to use on machine with python2 intact Message-ID: <475240BB-8DF9-4E90-968C-430846544A9C@bstage.com> Python noob here. I have a machine that I want to run borg on. this machine also has a few web apps which require python2. Borg requires python3. At first I installed python3 also and off I went, but after a while I think I was doing things wrong because I probably wasn?t keeping track of whether I was working with python2 or python3 when I ran pip to install extension libraries and such. things got messed up somehow in a way that I?m not knowledgable enough to explain or fix.. So..I am guessing there are ?proper? ways to work with both python2 and python3 on the same machine and I was hoping some of you could give me some advice about that, particularly as it relates to running borgbackup, which as of now is the only python3 thing I need to run, though I?m sure something else will come up eventually. 1 - I need the machine?s ?default? mode of operation to be python2.7 as I have many things installed which require that. typically this means that /usr/local/bin/python is sym linked to python2 by default, but I?d like to hear more about how to properly setup a machine with both 2 - I need to know what to do to install and use python3 properly so it won?t stomp over anything related to python2 3 - I need to know how to properly install extension libraries and things with pip so that I won?t accidentally install python2 things into the python3 enviornment or visa versa. Or does it matter? In other words, I guess I want to be able to install with pip into 2.7 place, or into 3.5 place?depending on which pip I call or something??? I?m not really sure what the right question is here... 4 - How does virtualenv play into all of this? thanks in advance From tw at waldmann-edv.de Mon Sep 25 21:51:38 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 26 Sep 2017 03:51:38 +0200 Subject: [Borgbackup] how to use on machine with python2 intact In-Reply-To: <475240BB-8DF9-4E90-968C-430846544A9C@bstage.com> References: <475240BB-8DF9-4E90-968C-430846544A9C@bstage.com> Message-ID: > So..I am guessing there are ?proper? ways to work with both python2 and python3 on the same machine Usually this is not a problem if you use the right python / python tool version as the software requires. E.g. when software wants python 2.x, you use pip, python (or better python2 or python2.7), etc. as commands / binary names. For python 3.x, you use pip3, python3 (or python3.X if you want a specific version), etc. It's quite the usual thing nowadays that the 3.x based tools have names ending with "3", while the old 2.x stuff has no such ending. It might depend a bit on the OS you are using though, some linux distributions already have boldly switched the default "python" to point to python3.x (symlink usually). In general, the stuff is kept separate, so there are no conflicts / overlap except for the non-versioned names. > running borgbackup, which as of now is the only python thing I need If you install a borgbackup package from your OS / linux distribution, this is usually taken care of. It will depend on python3.x etc. > though I?m sure something else will come up eventually. For sure, 2.x deadline is 2020. > 1 - I need the machine?s ?default? mode of operation to be python2.7 as I have many things installed which require that. typically this means that /usr/local/bin/python is sym linked to python2 by default, but I?d like to hear more about how to properly setup a machine with both That sounds like you are compiling python yourself. In that case, be careful, if you run "make install", it will replace your "python" link. "make altinstall" is the right way, it does not overwrite "python". > 2 - I need to know what to do to install and use python3 properly so it won?t stomp over anything related to python2 Just always use the right version of the python tools (like pip vs pip3). When creating virtualenv, you can give --python=... to specify the python version for the env. If your OS does not come with py3, you can also use the borgbackup binary, it includes python3.x and all libraries needed and won't overwrite / influence anything else on your system. https://github.com/borgbackup/borg/releases Cheers, Thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From steve at bstage.com Tue Sep 26 11:30:38 2017 From: steve at bstage.com (Steve Schow) Date: Tue, 26 Sep 2017 09:30:38 -0600 Subject: [Borgbackup] how to use on machine with python2 intact In-Reply-To: References: <475240BB-8DF9-4E90-968C-430846544A9C@bstage.com> Message-ID: <24ACBE31-DEC0-4A47-895A-814DF00D99C2@bstage.com> thank you so much for your reply and general tips about having both python2 and python3 on the same machine. Remembering to use pip3 for python3 expansion is the key I think. Interestingly, virtualenv, even after setting up an env with ?python=python3, installs into the env the 9.0.1 version of pip, which I *think* is for python2, but not sure. When I type pip2 -V I see a different version. Anyway... So I have borg ?almost? working? Not quite. Here are the details? I completely removed and reinstalled python totally from scratch on my machine, which is debian jessie. I actually went and found any and all python dirs I could find and nuked them all before installing python from scratch with apt. So ended up with python2.7 and python3.4. The binaries are under /usr/bin?. I also installed virtualenv I followed the borg installation instructions for debian to make sure I have all the needed packages installed, then I used the git install method, mainly because I want to use the ?patterns-from feature. All of that went fine, and I was able to source into a virtualenv, just like the borg install instructions and run borg? fine so far I inited a repo. check. Then I tried to ?create" a snapshot and got the following error: Local Exception Traceback (most recent call last): File "/home/sjs/src/borg/src/borg/archiver.py", line 3786, in main exit_code = archiver.run(args) File "/home/sjs/src/borg/src/borg/archiver.py", line 3714, in run return set_ec(func(args)) File "/home/sjs/src/borg/src/borg/archiver.py", line 148, in wrapper return method(self, args, repository=repository, **kwargs) File "/home/sjs/src/borg/src/borg/archiver.py", line 508, in do_create create_inner(archive, cache, fso) File "/home/sjs/src/borg/src/borg/archiver.py", line 465, in create_inner read_special=args.read_special, dry_run=dry_run, st=st) File "/home/sjs/src/borg/src/borg/archiver.py", line 568, in _process entries = helpers.scandir_inorder(path) File "/home/sjs/src/borg/src/borg/helpers/fs.py", line 117, in scandir_inorder return sorted(os.scandir(path), key=lambda dirent: dirent.inode()) AttributeError: 'module' object has no attribute 'scandir' Platform: Linux webman 4.4.79.x86_64.1 #1 SMP Mon Jul 31 15:14:40 PDT 2017 x86_64 Linux: debian 8.9 Borg: 1.2.0.dev96+g7c5a9d8 Python: CPython 3.4.2 PID: 32711 CWD: /home/sjs/src/goborg sys.argv: ['/home/sjs/src/goborg/ENV/bin/borg', 'create', '/data/raid/backup/borg/webman::{now}', '--compression', 'lz4', '--patterns-from', '/home/sjs/src/goborg/includes', '--list', '--stats', '--verbose', '--show-rc', '--filter=AME'] > On Sep 25, 2017, at 7:51 PM, Thomas Waldmann wrote: > >> So..I am guessing there are ?proper? ways to work with both python2 and python3 on the same machine > > Usually this is not a problem if you use the right python / python tool > version as the software requires. > > E.g. when software wants python 2.x, you use pip, python (or better > python2 or python2.7), etc. as commands / binary names. > > For python 3.x, you use pip3, python3 (or python3.X if you want a > specific version), etc. > > It's quite the usual thing nowadays that the 3.x based tools have names > ending with "3", while the old 2.x stuff has no such ending. > > It might depend a bit on the OS you are using though, some linux > distributions already have boldly switched the default "python" to point > to python3.x (symlink usually). > > In general, the stuff is kept separate, so there are no conflicts / > overlap except for the non-versioned names. > >> running borgbackup, which as of now is the only python thing I need > > If you install a borgbackup package from your OS / linux distribution, > this is usually taken care of. It will depend on python3.x etc. > >> though I?m sure something else will come up eventually. > > For sure, 2.x deadline is 2020. >> 1 - I need the machine?s ?default? mode of operation to be python2.7 as I have many things installed which require that. typically this means that /usr/local/bin/python is sym linked to python2 by default, but I?d like to hear more about how to properly setup a machine with both > > That sounds like you are compiling python yourself. > > In that case, be careful, if you run "make install", it will replace > your "python" link. > > "make altinstall" is the right way, it does not overwrite "python". > >> 2 - I need to know what to do to install and use python3 properly so it won?t stomp over anything related to python2 > > Just always use the right version of the python tools (like pip vs pip3). > > When creating virtualenv, you can give --python=... to specify the > python version for the env. > > If your OS does not come with py3, you can also use the borgbackup > binary, it includes python3.x and all libraries needed and won't > overwrite / influence anything else on your system. > > https://github.com/borgbackup/borg/releases > > Cheers, Thomas > > -- > > GPG ID: 9F88FB52FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Tue Sep 26 11:36:28 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Tue, 26 Sep 2017 17:36:28 +0200 Subject: [Borgbackup] how to use on machine with python2 intact In-Reply-To: <24ACBE31-DEC0-4A47-895A-814DF00D99C2@bstage.com> References: <475240BB-8DF9-4E90-968C-430846544A9C@bstage.com> <24ACBE31-DEC0-4A47-895A-814DF00D99C2@bstage.com> Message-ID: <8f4c0a7d-4160-3b23-eb10-e2a9cd7f04fc@waldmann-edv.de> > AttributeError: 'module' object has no attribute 'scandir' > > Platform: Linux webman 4.4.79.x86_64.1 #1 SMP Mon Jul 31 15:14:40 PDT 2017 x86_64 If you use master branch, python 3.5 is the minimum requirement. Using master for production is not recommended. Maybe you rather want to use 1.1-maint branch (minimum python 3.4) or a 1.1 release candidate binary (includes python). -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From maurice.libes at osupytheas.fr Wed Sep 27 09:05:44 2017 From: maurice.libes at osupytheas.fr (Maurice Libes) Date: Wed, 27 Sep 2017 15:05:44 +0200 Subject: [Borgbackup] hash function and collisions? Message-ID: hi to all Just a theoritical question that a friend of mine asked to me :-) I am not a mathematician and I dont know what to answer about the hash function and collision probability If I understand, in order to store the chunks and to reference them, a hash value is computed on the chunk data, with a hmac-sha256 function ? I read somewhere that the probability of having a collision is something less than 2^256 ? but what if it happens ? is borg able to detect a possible hash collision ( "n" fragments referenced by a same hash value) ? if not, this would say that we might have 2 differents chunks referenced by the same hash value and so a wrong file in the repository? sorry if the question makes no sense thanks for any comment M -- M. LIBES Service Informatique OSU Pytheas - UMS 3470 CNRS Batiment Oceanomed Campus de Luminy 13288 Marseille cedex 9 Tel: 04860 90529 -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2935 bytes Desc: Signature cryptographique S/MIME URL: From tw at waldmann-edv.de Wed Sep 27 09:46:29 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 27 Sep 2017 15:46:29 +0200 Subject: [Borgbackup] hash function and collisions? In-Reply-To: References: Message-ID: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> > Just a theoritical question that a friend of mine asked to me :-) > > I am not a mathematician and I dont know what to answer about the hash > function and collision probability The short answer is "you don't have to worry about that". Longer answer: https://github.com/borgbackup/borg/issues/170 -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From nift at maclisp.org Wed Sep 27 17:08:03 2017 From: nift at maclisp.org (Niels) Date: Wed, 27 Sep 2017 23:08:03 +0200 Subject: [Borgbackup] Backup fails with -> ValueError: time data '2017-09-11T23:54:21' does not match format '%Y-%m-%dT%H:%M:%S.%f' Message-ID: <87y3oz26ks.fsf@mojo.lan> Hi list I get this error on one of my repositories: Traceback (most recent call last): File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 2168, in main exit_code = archiver.run(args) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 2104, in run return set_ec(func(args)) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 107, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 331, in do_create create_inner(archive, cache) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 304, in create_inner read_special=args.read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 361, in _process status = archive.process_file(path, st, cache, self.ignore_inode) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 722, in process_file self.add_item(item) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 295, in add_item self.write_checkpoint() File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 299, in write_checkpoint self.save(self.checkpoint_name) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 333, in save self.manifest.write() File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/helpers.py", line 311, in write prev_ts = datetime.strptime(self.timestamp, "%Y-%m-%dT%H:%M:%S.%f") File "/usr/lib/python3.5/_strptime.py", line 510, in _strptime_datetime tt, fraction = _strptime(data_string, format) File "/usr/lib/python3.5/_strptime.py", line 343, in _strptime (data_string, format)) ValueError: time data '2017-09-11T23:54:21' does not match format '%Y-%m-%dT%H:%M:%S.%f' Platform: Linux desktop 4.9.0-3-amd64 #1 SMP Debian 4.9.30-2+deb9u3 (2017-08-06) x86_64 Linux: debian 9.1 Borg: 1.0.11 Python: CPython 3.5.3 PID: 16352 CWD: /home/niels/borg sys.argv: ['/usr/local/bin/borg', 'create', '-v', '--debug', '--stats', '--progress', '--compression', 'zlib,6', '--one-file-system', 'niels at nas:/backups/borg::bigdisk-{now:%Y-%m-%dT%H:%M:%S}', '/media/niels/bigdisk'] SSH_ORIGINAL_COMMAND: None Any ideas on how to fix/debug this? br Niels From steve at bstage.com Wed Sep 27 17:27:15 2017 From: steve at bstage.com (Steve Schow) Date: Wed, 27 Sep 2017 15:27:15 -0600 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> Message-ID: <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> I have recently ran some tests and notice that I got 25% faster performance by sending data from borg to a remote machine via SMB share vs using SSH remote tunnel. I also note that CPU performance on the source machine is substantially worse with the SSH tunnel, as the SSH process itself imposes signficant overhead in addition to the borg process. Can anyone comment about why this might be, how it might be improved or any particular reasons why the SSH tunnel would be preferable to saving the archive over SMB networking? Test scenario: Borg 1.1 source machine: readynas 6.8.0 dest machine: OSX 10.9.5 They are connected via gigabit switch With SMB, getting about 20MB/sec. Via SSH getting 15MB/sec readynas is mounting SMB share located on mac computer for SMB case. readynas is sending to mac via SSH, where borg comes up in server mode. In general, the NAS is capable of much more than 20MB/sec, as is the mac, so I am also wondering what else I might be able to do to speed things up. Ought to be able to get 50MB/sec From public at enkore.de Wed Sep 27 17:39:52 2017 From: public at enkore.de (Marian Beermann) Date: Wed, 27 Sep 2017 23:39:52 +0200 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> Message-ID: <5ee53279-4f93-2aa8-72da-fd638e820535@enkore.de> SMB 2 doesn't support encryption. On 27.09.2017 23:27, Steve Schow wrote: > I have recently ran some tests and notice that I got 25% faster performance by sending data from borg to a remote machine via SMB share vs using SSH remote tunnel. I also note that CPU performance on the source machine is substantially worse with the SSH tunnel, as the SSH process itself imposes signficant overhead in addition to the borg process. > > Can anyone comment about why this might be, how it might be improved or any particular reasons why the SSH tunnel would be preferable to saving the archive over SMB networking? > > Test scenario: > > Borg 1.1 > source machine: readynas 6.8.0 > dest machine: OSX 10.9.5 > They are connected via gigabit switch > > With SMB, getting about 20MB/sec. Via SSH getting 15MB/sec > > readynas is mounting SMB share located on mac computer for SMB case. > readynas is sending to mac via SSH, where borg comes up in server mode. > > In general, the NAS is capable of much more than 20MB/sec, as is the mac, so I am also wondering what else I might be able to do to speed things up. Ought to be able to get 50MB/sec > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From public at enkore.de Wed Sep 27 17:40:32 2017 From: public at enkore.de (Marian Beermann) Date: Wed, 27 Sep 2017 23:40:32 +0200 Subject: [Borgbackup] Backup fails with -> ValueError: time data '2017-09-11T23:54:21' does not match format '%Y-%m-%dT%H:%M:%S.%f' In-Reply-To: <87y3oz26ks.fsf@mojo.lan> References: <87y3oz26ks.fsf@mojo.lan> Message-ID: https://github.com/borgbackup/borg/issues/2994#issuecomment-331842939 From steve at bstage.com Wed Sep 27 18:45:38 2017 From: steve at bstage.com (Steve Schow) Date: Wed, 27 Sep 2017 16:45:38 -0600 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <5ee53279-4f93-2aa8-72da-fd638e820535@enkore.de> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> <5ee53279-4f93-2aa8-72da-fd638e820535@enkore.de> Message-ID: <746A9A72-D9D9-445A-83B3-DB4382B6C095@bstage.com> Right.. when you say it doesn?t support encryption, I assume that borg itself is encrypting the archive anyway right? I?m actually using ?encryption=none anyway because its all on my local LAN and I don?t care about my archive being encrypted?but just for the sake of understanding the situation?. Plus I?m actually using SMB1/CIF general question?if you have borg using encryption??and also using SSH?then isn?t it getting encrypted twice? > On Sep 27, 2017, at 3:39 PM, Marian Beermann wrote: > > SMB 2 doesn't support encryption. > > On 27.09.2017 23:27, Steve Schow wrote: >> I have recently ran some tests and notice that I got 25% faster performance by sending data from borg to a remote machine via SMB share vs using SSH remote tunnel. I also note that CPU performance on the source machine is substantially worse with the SSH tunnel, as the SSH process itself imposes signficant overhead in addition to the borg process. >> >> Can anyone comment about why this might be, how it might be improved or any particular reasons why the SSH tunnel would be preferable to saving the archive over SMB networking? >> >> Test scenario: >> >> Borg 1.1 >> source machine: readynas 6.8.0 >> dest machine: OSX 10.9.5 >> They are connected via gigabit switch >> >> With SMB, getting about 20MB/sec. Via SSH getting 15MB/sec >> >> readynas is mounting SMB share located on mac computer for SMB case. >> readynas is sending to mac via SSH, where borg comes up in server mode. >> >> In general, the NAS is capable of much more than 20MB/sec, as is the mac, so I am also wondering what else I might be able to do to speed things up. Ought to be able to get 50MB/sec >> >> >> >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup >> > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From jost+lists at dimejo.at Thu Sep 28 02:18:07 2017 From: jost+lists at dimejo.at (Alex JOST) Date: Thu, 28 Sep 2017 08:18:07 +0200 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <746A9A72-D9D9-445A-83B3-DB4382B6C095@bstage.com> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> <5ee53279-4f93-2aa8-72da-fd638e820535@enkore.de> <746A9A72-D9D9-445A-83B3-DB4382B6C095@bstage.com> Message-ID: <0a8d097b-5690-02af-802e-34a0231bf349@dimejo.at> Am 28.09.2017 um 00:45 schrieb Steve Schow: > Right.. when you say it doesn?t support encryption, I assume that borg itself is encrypting the archive anyway right? > > I?m actually using ?encryption=none anyway because its all on my local LAN and I don?t care about my archive being encrypted?but just for the sake of understanding the situation?. > > Plus I?m actually using SMB1/CIF > > general question?if you have borg using encryption??and also using SSH?then isn?t it getting encrypted twice? Borg encrypts the data itself. Data is encrypted at the source and saved as is at the destination. This protects from an attacker trying to read the data at the destination. SSH encrypts the transport channel. The data stream is encrypted on the source and decrypted at the destination. This protects from an attacker sitting in between source and destination (MITM - man in the middle). -- Alex JOST From steve at bstage.com Thu Sep 28 15:38:16 2017 From: steve at bstage.com (Steve Schow) Date: Thu, 28 Sep 2017 13:38:16 -0600 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> Message-ID: <97B8BE65-EDE5-4BF2-BD76-480E48ACEB8B@bstage.com> question: When using remote backup server?does the remote server itself offload any of the deduplication processing? Would that be a compelling reason to use remote over SSH vs directly over a SMB share? > On Sep 27, 2017, at 3:27 PM, Steve Schow wrote: > > I have recently ran some tests and notice that I got 25% faster performance by sending data from borg to a remote machine via SMB share vs using SSH remote tunnel. I also note that CPU performance on the source machine is substantially worse with the SSH tunnel, as the SSH process itself imposes signficant overhead in addition to the borg process. > > Can anyone comment about why this might be, how it might be improved or any particular reasons why the SSH tunnel would be preferable to saving the archive over SMB networking? > > Test scenario: > > Borg 1.1 > source machine: readynas 6.8.0 > dest machine: OSX 10.9.5 > They are connected via gigabit switch > > With SMB, getting about 20MB/sec. Via SSH getting 15MB/sec > > readynas is mounting SMB share located on mac computer for SMB case. > readynas is sending to mac via SSH, where borg comes up in server mode. > > In general, the NAS is capable of much more than 20MB/sec, as is the mac, so I am also wondering what else I might be able to do to speed things up. Ought to be able to get 50MB/sec > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From tw at waldmann-edv.de Thu Sep 28 15:46:31 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 28 Sep 2017 21:46:31 +0200 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <97B8BE65-EDE5-4BF2-BD76-480E48ACEB8B@bstage.com> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> <97B8BE65-EDE5-4BF2-BD76-480E48ACEB8B@bstage.com> Message-ID: On 09/28/2017 09:38 PM, Steve Schow wrote: > question: When using remote backup server?does the remote server itself offload any of the deduplication processing? No, not that. The point of running client / server (== two borg processes talking via ssh to each other) is that you have an agent on the server. Thus, repository operations are local the the server side agent. That speeds up: - repo check (local operation) - starting a transaction (does not need to copy repo index over network fs, but can do it locally) - likely also other operations, because the server can do repo fs access and just deliver the result - instead of the client doing that over network fs in a likely less efficient way. Also, depending on your network filesystem: - seeks and partial reads will work (they also work for a normal network fs, but likely not for some of the cloud-backed fs) -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 From steve at bstage.com Thu Sep 28 16:26:20 2017 From: steve at bstage.com (Steve Schow) Date: Thu, 28 Sep 2017 14:26:20 -0600 Subject: [Borgbackup] Fwd: remove ssh or via SMB? References: Message-ID: So you?re saying the repo index has to be pushed back to the client, over SMB, while with the SSH server it is just kept in the server. that?s no problem and a reason to use SSH, but I am curious for this discussion more about the network traffic in the other direction?from client to server. Does anything end up getting sent twice (or more) over SMB that wouldn?t otherwise get sent in that direction over SSH? Again, I?m just trying to determine which method is more efficient here. I see 30% more network bandwidth in the client->server direction during create over SMB?and sustained that way for hours? Which means I can either deduce that SMB is transferring the data more efficiently??30% faster?.or? if its doing duplicated transfers in that direction?then that could show the increased bandwidth but would not mean 30% faster end result. > From: Thomas Waldmann > Subject: Re: [Borgbackup] remove ssh or via SMB? > Date: September 28, 2017 at 2:09:05 PM MDT > To: Steve Schow > > ssh of course needs some more cpu for the transport encryption. > > if you just did a quick little test, you didn't have a big repo index, > so starting a transaction was cheap in both cases. > > that will look different if you have a repo index of e.g. 1GB, then it > will have to copy that at the beginning of each transaction (at the > start of the backup and at every checkpoint it is doing). > > copying 1GB locally is no big issue, but pumping it back and forth over > the network usually takes a bit longer. > > -- > > GPG ID: 9F88FB52FAF7B393 > GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393 > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jungleboogie0 at gmail.com Thu Sep 28 18:57:20 2017 From: jungleboogie0 at gmail.com (jungle Boogie) Date: Thu, 28 Sep 2017 15:57:20 -0700 Subject: [Borgbackup] remove ssh or via SMB? In-Reply-To: <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> References: <7482059c-7028-f22b-3deb-6f5b56ae562a@waldmann-edv.de> <88046CA6-468E-4B20-84BA-305520BB24CB@bstage.com> Message-ID: On 27 September 2017 at 14:27, Steve Schow wrote: > I have recently ran some tests and notice that I got 25% faster performance by sending data from borg to a remote machine via SMB share vs using SSH remote tunnel. I also note that CPU performance on the source machine is substantially worse with the SSH tunnel, as the SSH process itself imposes signficant overhead in addition to the borg process. > > Can anyone comment about why this might be, how it might be improved or any particular reasons why the SSH tunnel would be preferable to saving the archive over SMB networking? > As others have said, it's likely the encryption taking place in ssh. As you've already encrypted the data, maybe you can use HPN-SSH and see what your results are like: https://www.psc.edu/index.php/hpn-ssh & https://sourceforge.net/p/hpnssh/wiki/Home HPN-SSH is a series of patches for OpenSSH to provide improved performance and functionality. The core of HPN-SSH is the implementation of a dynamic receive side buffer. By allowing this buffer to grow with the TCP receive buffer it's possible to attain significant throughput improvement - up to two orders of magnitude depending on the network path. Patch sets for each OpenSSH release will be found in the appropriate subfolder. From nift at maclisp.org Fri Sep 29 08:38:25 2017 From: nift at maclisp.org (Niels) Date: Fri, 29 Sep 2017 14:38:25 +0200 Subject: [Borgbackup] Backup fails with -> ValueError: time data '2017-09-11T23:54:21' does not match format '%Y-%m-%dT%H:%M:%S.%f' In-Reply-To: (Marian Beermann's message of "Wed, 27 Sep 2017 23:40:32 +0200") References: <87y3oz26ks.fsf@mojo.lan> Message-ID: <87o9pt1xz2.fsf@mojo.lan> Marian Beermann writes: > https://github.com/borgbackup/borg/issues/2994#issuecomment-331842939 Thanks for the pointer. I've removed the ".%f" in helpers.py as described in the issue, on both the client and server. But unfortunately I get another error: Traceback (most recent call last): File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 2168, in main exit_code = archiver.run(args) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 2104, in run return set_ec(func(args)) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 107, in wrapper return method(self, args, repository=repository, **kwargs) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 331, in do_create create_inner(archive, cache) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 304, in create_inner read_special=args.read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 380, in _process read_special=read_special, dry_run=dry_run) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archiver.py", line 361, in _process status = archive.process_file(path, st, cache, self.ignore_inode) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 722, in process_file self.add_item(item) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 295, in add_item self.write_checkpoint() File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 299, in write_checkpoint self.save(self.checkpoint_name) File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/archive.py", line 333, in save self.manifest.write() File "/usr/local/src/borg-env/lib/python3.5/site-packages/borg/helpers.py", line 311, in write prev_ts = datetime.strptime(self.timestamp, "%Y-%m-%dT%H:%M:%S") File "/usr/lib/python3.5/_strptime.py", line 510, in _strptime_datetime tt, fraction = _strptime(data_string, format) File "/usr/lib/python3.5/_strptime.py", line 346, in _strptime data_string[found.end():]) ValueError: unconverted data remains: .061399 Niels From tw at waldmann-edv.de Sat Sep 30 23:11:14 2017 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sun, 1 Oct 2017 05:11:14 +0200 Subject: [Borgbackup] borgbackup 1.1.0 release candidate 4 released Message-ID: <82f64e6b-aeb6-5ffd-d63b-8c5b60e689f2@waldmann-edv.de> https://github.com/borgbackup/borg/releases/tag/1.1.0rc4 please help testing and provide feedback. details see url. cheers, thomas -- GPG ID: 9F88FB52FAF7B393 GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393