From tschoening at am-soft.de Sun Apr 2 13:19:39 2023 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Sun, 2 Apr 2023 19:19:39 +0200 Subject: [Borgbackup] lzma, 3-2-1 rule In-Reply-To: References: Message-ID: <415909680.20230402191939@am-soft.de> Guten Tag Boris Kirkorowicz, am Freitag, 31. M?rz 2023 um 11:36 schrieben Sie: > This is new to me, maybe I overlooked something when choosing "-C > lzma,9". You most likely missed the following warning: > Giving levels above 6 is pointless and counterproductive because it > does not compress better due to the buffer size used by borg - but > it wastes lots of CPU cycles and RAM. https://manpages.debian.org/testing/borgbackup/borg-compression.1.en.html > So should I stop backing up using lzma and start again with an other > compression method, or is it safe to continue? I don't think that reading LZMA will be removed too soon, using it for new data might be. OTOH, different compression methods can be mixed freely, so you don't need to care too much: > It is no problem to mix different compression methods in one > repo[...] https://manpages.debian.org/testing/borgbackup/borg-compression.1.en.html There was a thread recently about compression performance, regarding that you should definitely stop using LZMA for performance reasons. https://mail.python.org/pipermail/borgbackup/2023q1/002165.html > Second question: the article says that it is no good idea to > fulfill the 3-2-1 rule by copying the borg repositories, since > errors might be copied. This sounds plausible, but on the other hand > it would multiply the time of backup. In my case it could take more > than one year, so I wonder how to do this. Any advice how to handle this? Don't waste backup time with LZMA anymore and run multiple individual backups to different repos one after another. Besides that and depending on the performance of your source system, you might run multiple BORG processes concurrently as long as they use different target repos. If I remember correctly you discussed that for your initial backup yourself already. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz -------------- next part -------------- A non-text attachment was scrubbed... Name: BORBACKUP_COMPRESSION_TESTS.explic Type: application/octet-stream Size: 7051 bytes Desc: not available URL: From bkborg at kirk.de Mon Apr 3 06:59:28 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Mon, 3 Apr 2023 12:59:28 +0200 Subject: [Borgbackup] lzma, 3-2-1 rule In-Reply-To: <415909680.20230402191939@am-soft.de> References: <415909680.20230402191939@am-soft.de> Message-ID: <0d9b461c-503f-b0a4-f640-a85783ddf620@kirk.de> Hi, Am 02.04.23 um 19:19 schrieb Thorsten Sch?ning: > Guten Tag Boris Kirkorowicz, > am Freitag, 31. M?rz 2023 um 11:36 schrieben Sie: > >> This is new to me, maybe I overlooked something when choosing "-C >> lzma,9". > > You most likely missed the following warning: > >> Giving levels above 6 is pointless and counterproductive because it >> does not compress better due to the buffer size used by borg - but >> it wastes lots of CPU cycles and RAM. to explain how I got to chose lzma,9: I did some testings by my own with a sample dir tree (7,5 GB) to compare compression methods. > Compression Time Repo Size Ratio > lz4 00:51:55 5,8 GB 77?% > zlib,6 00:43:31 4,4 GB 59?% > lzma,1 00:51:51 4,5 GB 60?% > lzma,6 01:02:50 4,2 GB 56?% > lzma,9 01:00:11 4,2 GB 56?% In my sample, lzma,9 was slightly faster than lzma,6 without any other difference, and it compressed better than the other tested methods. Unfortunately, I did not test all compression methods, just these above. Since I plan so transfer the repos to a remote site, I preferred the best compression ratio to save bandwidth. Additionally, I logged the processor load and found that the load is mostly at 60~80% with rare peaks to 100%. Since the server has nothing else to do during backup, this seems to me as a practicable setting. > I don't think that reading LZMA will be removed too soon, using it for > new data might be. OTOH, different compression methods can be mixed > freely, so you don't need to care too much: I need to save the data for at least 10 years, so it is important to be sure that I can read the backups after that long time. Every period below 10 years would be too soon for me. What do you think: does lzma survive this period, at least reading? >> It is no problem to mix different compression methods in one >> repo[...] > > https://manpages.debian.org/testing/borgbackup/borg-compression.1.en.html > > There was a thread recently about compression performance, regarding > that you should definitely stop using LZMA for performance reasons. OK, I'll complete my testings with the other compression methods to pick another one. >> Second question: the article says that it is no good idea to >> fulfill the 3-2-1 rule by copying the borg repositories, since >> errors might be copied. This sounds plausible, but on the other hand >> it would multiply the time of backup. In my case it could take more >> than one year, so I wonder how to do this. Any advice how to handle this? > > Don't waste backup time with LZMA anymore and run multiple individual > backups to different repos one after another. Besides that and > depending on the performance of your source system, you might run > multiple BORG processes concurrently as long as they use different > target repos. If I remember correctly you discussed that for your > initial backup yourself already. That's what I do: I split the data into reasonable trees and invoke borg for each tree in parallel. So every borg process runs in e separate core. I did my very best, but three of these trees remain that large that it takes so much time. I'll have a second look at it, but I don't expect to find big improvements. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From tschoening at am-soft.de Mon Apr 3 08:15:22 2023 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Mon, 3 Apr 2023 14:15:22 +0200 Subject: [Borgbackup] lzma, 3-2-1 rule In-Reply-To: <0d9b461c-503f-b0a4-f640-a85783ddf620@kirk.de> References: <415909680.20230402191939@am-soft.de> <0d9b461c-503f-b0a4-f640-a85783ddf620@kirk.de> Message-ID: <413054155.20230403141522@am-soft.de> Guten Tag Boris Kirkorowicz, am Montag, 3. April 2023 um 12:59 schrieben Sie: > I need to save the data for at least 10 years, so it is important > to be sure that I can read the backups after that long time. Every > period below 10 years would be too soon for me. What do you think: > does lzma survive this period, at least reading? Yes, but in the end nobody knows anyway. If it's that important for you, you need to backup the backup software, a compatible host being able to execute it etc. anyway as well. Which makes LZMA itself less of a problem. > That's what I do: I split the data into reasonable trees and invoke > borg for each tree in parallel.[...] That's not what I was talking about, but you can read the same data concurrently with Borg as well and backup into different repos. So you can actually execute standard and off-site backup or standard backup with multiple different target media at the same time. Especially in case of modern CoW file systems like BTRFS and ZFS after simply taking two snapshots one after another and basing the backups individually on each of them. Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From felix.schwarz at oss.schwarz.eu Tue Apr 4 16:38:59 2023 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Tue, 4 Apr 2023 22:38:59 +0200 Subject: [Borgbackup] lzma, 3-2-1 rule In-Reply-To: References: Message-ID: Am 31.03.23 um 11:36 schrieb Boris Kirkorowicz: > Second question: the article says that it is no good idea to fulfill the 3-2-1 > rule by copying the borg repositories, since errors might be copied. This sounds > plausible, but on the other hand it would multiply the time of backup. In my > case it could take more than one year, so I wonder how to do this. Any advice > how to handle this? Here is what I am doing (with much smaller data sizes): - backups are stored on my NAS - the borgbackup files are synced to remote office location (copy) I'm not so much concerned about *preventing* propagating errors - I rather focus on *detection*. borg can verify the integrity of backup repositories so I'm running "borg check" on the remote location regularly. The good thing is that the check only needs the remote location (no strain on production resources by doing the backup twice) and if the check succeeds I can also trust the backups on my NAS should be ok. Maybe that approach can work for you as well? Felix From tw at waldmann-edv.de Sat Apr 15 09:34:27 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 15 Apr 2023 15:34:27 +0200 Subject: [Borgbackup] lzma, 3-2-1 rule In-Reply-To: References: Message-ID: <536032a7-382d-d74a-5d56-f9f8af0c2abd@waldmann-edv.de> > in the german computer magazine c't, I found an article about borgbackup > -very nice. There it is mentioned that the compression method lzma is > deprecated and only left for compatibility reasons. Hmm, IIRC our docs rather say zlib/lzma are "legacy" (but not deprecated)? Legacy meaning they are older algorithms and maybe not as nice/efficient/fast/versatile as zstd (or lz4, if one wants highest possible speed). Anyway, consider zlib and lzma are provided by python stdlib, removing them from borg would not give us any real "advantage" (like saving some dependency), so guess they are not going away for a long time. Removing any compression algorithm is problematic anyway as there might be users having a ton of data compressed with it (like in your case). So the only points in time we could do that is a breaking release (like borg2) that needs "transferring" archives from an old repo to a new one anyway. For borg2, the is no plan yet to remove any compression algorithm. > My concern: my actual backup is running since February every night, and > the estimated completion might be in June or July Oh, wow. O.O > and the constantly slowing down backup rate of now ~5 > GB/h compressed/deduplicated data. You checked that borg does not use more RAM than you have (neither on client nor or server side)? From bkborg at kirk.de Tue May 9 04:07:44 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Tue, 9 May 2023 10:07:44 +0200 Subject: [Borgbackup] recommended check option Message-ID: Hi, the backup job running since January has finished now. I split the data to multiple repos. Two of the repos ended with rc=1: > terminating with warning status, rc 1 I guess this is due to an interruption of the IP connect to the backup disks yesterday: > [...] > /home/boris/public_html/.directory: read: [Errno 107] Transport endpoint is not connected > Local Exception > Traceback (most recent call last): > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 183, in wrapper > return method(self, args, repository=repository, **kwargs) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 675, in do_create > create_inner(archive, cache, fso) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 629, in create_inner > archive.save(comment=args.comment, timestamp=args.timestamp) > File "/usr/lib64/python3.10/site-packages/borg/archive.py", line 613, in save > self.items_buffer.flush(flush=True) > File "/usr/lib64/python3.10/site-packages/borg/archive.py", line 385, in flush > self.chunks.append(self.write_chunk(chunk)) > File "/usr/lib64/python3.10/site-packages/borg/archive.py", line 401, in write_chunk > id_, _, _ = self.cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats, wait=False) > File "/usr/lib64/python3.10/site-packages/borg/cache.py", line 951, in add_chunk > self.repository.put(id, data, wait=wait) > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1230, in put > segment, offset = self.io.write_put(id, data) > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1643, in write_put > fd.write(b''.join((crc, header, id, data))) > File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 174, in write > self.f.write(data) > OSError: [Errno 107] Transport endpoint is not connected > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 194, in close > self.sync() > File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 182, in sync > self.f.flush() > OSError: [Errno 107] Transport endpoint is not connected > > During handling of the above exception, another exception occurred: > > OSError: [Errno 107] Transport endpoint is not connected > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5213, in main > exit_code = archiver.run(args) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5144, in run > return set_ec(func(args)) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 168, in wrapper > with repository: > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 214, in __exit__ > self.close() > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 476, in close > self.io.close() > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1313, in close > self.close_segment() > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1468, in close_segment > fd.close() > File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 196, in close > self.f.close() > OSError: [Errno 107] Transport endpoint is not connected > > Platform: Linux sfo2205 6.1.7-1-default #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c) x86_64 > Linux: Unknown Linux > Borg: 1.2.4 Python: CPython 3.10.10 msgpack: 1.0.4 fuse: pyfuse3 3.2.2 [pyfuse3,llfuse] > PID: 29494 CWD: /root > sys.argv: ['/usr/bin/borg', 'create', '-v', '--stats', '--show-rc', '-C', 'lzma,6', '/mnt/rs1219a/Boris::{now:%Y-%m-%d_%H-%M}', '/home/boris'] > SSH_ORIGINAL_COMMAND: None > > terminating with error status, rc 2 > Exception ignored in: > Traceback (most recent call last): > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 189, in __del__ > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 476, in close > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1314, in close > File "/usr/lib64/python3.10/site-packages/borg/lrucache.py", line 50, in clear > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1320, in _close_fd > OSError: [Errno 107] Transport endpoint is not connected > 2023-05-08_14-13-19: Datensicherung Nr. 3 mit borgbackup beendet (2) mit RC: 2 Now I wonder what is to do best. First thing might be borg check /mnt/rs1219a/Boris but there are some options mentioned in the docs that I do not understand, resp. their meaning. What is the recommended procedure here? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From bkborg at kirk.de Wed May 10 06:31:07 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Wed, 10 May 2023 12:31:07 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: References: Message-ID: Hi, first, I tried with one of the smaller repos that were not affected: > # borg check --repository-only /mnt/rs1219a/usrlocal/ > Failed to create/acquire the lock /mnt/rs1219a/usrlocal/lock (timeout). It is the same with all repos. The repos are not encrypted. So I wonder what is meant with the failing lock. All I found was break-lock, and so I tried > # borg break-lock /mnt/rs1219a/usrlocal/ > # borg -v check --repository-only /mnt/rs1219a/usrlocal/ > Starting repository check > Local Exception > Traceback (most recent call last): > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5213, in main > exit_code = archiver.run(args) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5144, in run > return set_ec(func(args)) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 183, in wrapper > return method(self, args, repository=repository, **kwargs) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 343, in do_check > if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration): > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1026, in check > self.save_config(self.path, self.config) > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 332, in save_config > secure_erase(old_config_path, avoid_collateral_damage=True) > File "/usr/lib64/python3.10/site-packages/borg/helpers/fs.py", line 199, in secure_erase > with open(path, 'r+b') as fd: > OSError: [Errno 5] Input/output error: '/mnt/rs1219a/usrlocal/config.old' > > Platform: Linux sfo2205 6.1.7-1-default #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c) x86_64 > Linux: Unknown Linux > Borg: 1.2.4 Python: CPython 3.10.10 msgpack: 1.0.4 fuse: pyfuse3 3.2.2 [pyfuse3,llfuse] > PID: 18561 CWD: /root > sys.argv: ['/usr/bin/borg', '-v', 'check', '--repository-only', '/mnt/rs1219a/usrlocal/'] > SSH_ORIGINAL_COMMAND: None I guess I need some assistance here. Am 09.05.23 um 10:07 schrieb Boris Kirkorowicz: > Hi, > the backup job running since January has finished now. > > I split the data to multiple repos. Two of the repos ended with rc=1: >> terminating with warning status, rc 1 > > I guess this is due to an interruption of the IP connect to the backup > disks yesterday: > >> [...] >> /home/boris/public_html/.directory: read: [Errno 107] Transport endpoint is not connected >> Local Exception >> Traceback (most recent call last): >> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 183, in wrapper >> return method(self, args, repository=repository, **kwargs) >> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 675, in do_create >> create_inner(archive, cache, fso) >> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 629, in create_inner >> archive.save(comment=args.comment, timestamp=args.timestamp) >> File "/usr/lib64/python3.10/site-packages/borg/archive.py", line 613, in save >> self.items_buffer.flush(flush=True) >> File "/usr/lib64/python3.10/site-packages/borg/archive.py", line 385, in flush >> self.chunks.append(self.write_chunk(chunk)) >> File "/usr/lib64/python3.10/site-packages/borg/archive.py", line 401, in write_chunk >> id_, _, _ = self.cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats, wait=False) >> File "/usr/lib64/python3.10/site-packages/borg/cache.py", line 951, in add_chunk >> self.repository.put(id, data, wait=wait) >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1230, in put >> segment, offset = self.io.write_put(id, data) >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1643, in write_put >> fd.write(b''.join((crc, header, id, data))) >> File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 174, in write >> self.f.write(data) >> OSError: [Errno 107] Transport endpoint is not connected >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 194, in close >> self.sync() >> File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 182, in sync >> self.f.flush() >> OSError: [Errno 107] Transport endpoint is not connected >> >> During handling of the above exception, another exception occurred: >> >> OSError: [Errno 107] Transport endpoint is not connected >> >> During handling of the above exception, another exception occurred: >> >> Traceback (most recent call last): >> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5213, in main >> exit_code = archiver.run(args) >> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5144, in run >> return set_ec(func(args)) >> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 168, in wrapper >> with repository: >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 214, in __exit__ >> self.close() >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 476, in close >> self.io.close() >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1313, in close >> self.close_segment() >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1468, in close_segment >> fd.close() >> File "/usr/lib64/python3.10/site-packages/borg/platform/base.py", line 196, in close >> self.f.close() >> OSError: [Errno 107] Transport endpoint is not connected >> >> Platform: Linux sfo2205 6.1.7-1-default #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c) x86_64 >> Linux: Unknown Linux >> Borg: 1.2.4 Python: CPython 3.10.10 msgpack: 1.0.4 fuse: pyfuse3 3.2.2 [pyfuse3,llfuse] >> PID: 29494 CWD: /root >> sys.argv: ['/usr/bin/borg', 'create', '-v', '--stats', '--show-rc', '-C', 'lzma,6', '/mnt/rs1219a/Boris::{now:%Y-%m-%d_%H-%M}', '/home/boris'] >> SSH_ORIGINAL_COMMAND: None >> >> terminating with error status, rc 2 >> Exception ignored in: >> Traceback (most recent call last): >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 189, in __del__ >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 476, in close >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1314, in close >> File "/usr/lib64/python3.10/site-packages/borg/lrucache.py", line 50, in clear >> File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 1320, in _close_fd >> OSError: [Errno 107] Transport endpoint is not connected >> 2023-05-08_14-13-19: Datensicherung Nr. 3 mit borgbackup beendet (2) mit RC: 2 > > > Now I wonder what is to do best. First thing might be > > borg check /mnt/rs1219a/Boris > > but there are some options mentioned in the docs that I do not > understand, resp. their meaning. What is the recommended procedure here? > > > > -- Mit freundlichem Gru? Best regards ? Kirkorowicz From lazyvirus at gmx.com Wed May 10 10:27:40 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Wed, 10 May 2023 16:27:40 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: References: Message-ID: <20230510162740.668c7cf2@msi.defcon1.lan> On Wed, 10 May 2023 12:31:07 +0200 Boris Kirkorowicz wrote: Hi, > first, I tried with one of the smaller repos that were not affected: > > # borg check --repository-only /mnt/rs1219a/usrlocal/ > > Failed to create/acquire the lock /mnt/rs1219a/usrlocal/lock > > (timeout). > > It is the same with all repos. The repos are not encrypted. So I > wonder what is meant with the failing lock. All I found was > break-lock, and so I tried > > > # borg break-lock /mnt/rs1219a/usrlocal/ > > # borg -v check --repository-only /mnt/rs1219a/usrlocal/ > > Starting repository check > > Local Exception > > Traceback (most recent call last): > > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line > > 5213, in main exit_code = archiver.run(args) > > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line > > 5144, in run return set_ec(func(args)) > > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line > > 183, in wrapper return method(self, args, repository=repository, > > **kwargs) File > > "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 343, > > in do_check if not repository.check(repair=args.repair, > > save_space=args.save_space, max_duration=args.max_duration): File > > "/usr/lib64/python3.10/site-packages/borg/repository.py", line > > 1026, in check self.save_config(self.path, self.config) File > > "/usr/lib64/python3.10/site-packages/borg/repository.py", line 332, > > in save_config secure_erase(old_config_path, > > avoid_collateral_damage=True) File > > "/usr/lib64/python3.10/site-packages/borg/helpers/fs.py", line 199, > > in secure_erase with open(path, 'r+b') as fd: OSError: [Errno 5] > > Input/output error: '/mnt/rs1219a/usrlocal/config.old' ^^^^^^^^^^ Apparently, you have a problem on the above file, very possibly a bad sector which renders it unreadable, this might be bad for the rest of the repo. Copying a bad file often works (I just have had the case), but this won't solve the main problem, as BB wants to read the bad one - at this point, my contention is BB wants to read any existing file there. This is a problem as it could have two origin: a random bad sector on a big disk, which needs to be fixed before removing the file to avoid to hit on a BB file in the future, or it can be your disk dying. Anyway, you'll have to find and fix the bad sector correctly, to either have it relocated or marked as bad in the FS. So, fix the bad sector, copy the repo on another disk (where you'll run the BB check) and do what's needed for the original disk (smart long test and formatting in ext4 with destructive patterns, which can take a looong time if the disk is big). I don't know for the rest as I only use NFS, which have the advantage of putting things on hold if the repo isn't available anymore (eg: reboot for some reason) and takes back where it was when the repo is back online. Jean-Yves From ContainerPi at disroot.org Wed May 10 10:34:42 2023 From: ContainerPi at disroot.org (Evelyn Pereira Souza) Date: Wed, 10 May 2023 16:34:42 +0200 Subject: [Borgbackup] S3 support? Message-ID: Hi after using rsync and s3cmd for a very long time, I switched to borgbackup a month ago. It works great, but I want an offsite upload to Amazon S3 and have not found anything regarding S3 support. I used to use s3cmd and gpg for AWS S3. kind regards Evelyn -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x61776FA8E38403FB.asc Type: application/pgp-keys Size: 2448 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 665 bytes Desc: OpenPGP digital signature URL: From tw at waldmann-edv.de Wed May 10 10:41:50 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 10 May 2023 16:41:50 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: References: Message-ID: <76b14203-d2ec-7972-4829-f6aca06a7670@waldmann-edv.de> Wow, that took long. Interrupted backup (connection issue, transport endpoint not connected): just start borg create again and again in the same way. It won't transmit the data again which it already has in the repo. For this, a "borg check" is not needed. I/O Error: fs or storage device malfunctioning, if you have access, run a SMART long test on the disks and check the SMART status/logs (smartctl). From bkborg at kirk.de Wed May 10 10:51:40 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Wed, 10 May 2023 16:51:40 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: <20230510162740.668c7cf2@msi.defcon1.lan> References: <20230510162740.668c7cf2@msi.defcon1.lan> Message-ID: Hello, Am 10.05.23 um 16:27 schrieb Bzzzz: > On Wed, 10 May 2023 12:31:07 +0200 > Boris Kirkorowicz wrote: > > Hi, > >> first, I tried with one of the smaller repos that were not affected: >>> # borg check --repository-only /mnt/rs1219a/usrlocal/ >>> Failed to create/acquire the lock /mnt/rs1219a/usrlocal/lock >>> (timeout). >> >> It is the same with all repos. The repos are not encrypted. So I >> wonder what is meant with the failing lock. All I found was >> break-lock, and so I tried >> >>> # borg break-lock /mnt/rs1219a/usrlocal/ >>> # borg -v check --repository-only /mnt/rs1219a/usrlocal/ >>> Starting repository check >>> Local Exception >>> Traceback (most recent call last): >>> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line >>> 5213, in main exit_code = archiver.run(args) >>> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line >>> 5144, in run return set_ec(func(args)) >>> File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line >>> 183, in wrapper return method(self, args, repository=repository, >>> **kwargs) File >>> "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 343, >>> in do_check if not repository.check(repair=args.repair, >>> save_space=args.save_space, max_duration=args.max_duration): File >>> "/usr/lib64/python3.10/site-packages/borg/repository.py", line >>> 1026, in check self.save_config(self.path, self.config) File >>> "/usr/lib64/python3.10/site-packages/borg/repository.py", line 332, >>> in save_config secure_erase(old_config_path, >>> avoid_collateral_damage=True) File >>> "/usr/lib64/python3.10/site-packages/borg/helpers/fs.py", line 199, >>> in secure_erase with open(path, 'r+b') as fd: OSError: [Errno 5] >>> Input/output error: '/mnt/rs1219a/usrlocal/config.old' > ^^^^^^^^^^ > Apparently, you have a problem on the above file, very possibly a bad > sector which renders it unreadable, this might be bad for the rest of > the repo. that seems not to be the case, since I can read it using cat and edit it using vi. The error remains the same. The text file contains some info about the repo: > [repository] > version = 1 > segments_per_dir = 1000 > max_segment_size = 524288000 > append_only = 0 > storage_quota = 0 > additional_free_space = 0 > id = 180ba330d0a627f2259f0ec4132fef7a45d135cf774b6c41ec75755744d5bdf9 Could the error be that e.g. the id string is wrong, or something similar? > This is a problem as it could have two origin: a random bad sector on a > big disk, which needs to be fixed before removing the file to avoid to > hit on a BB file in the future, or it can be your disk dying. The disks are brand new, initialized using ext4 just before running borg. > So, fix the bad sector, copy the repo on another disk (where you'll run > the BB check) and do what's needed for the original disk (smart long > test and formatting in ext4 with destructive patterns, which can take > a looong time if the disk is big). AFAICS it applies to all repos. The backup took about four months to complete, so formatting would not be that significant. So I hope that the cause is something different and it could be repaired by another method. > I don't know for the rest as I only use NFS, which have the advantage > of putting things on hold if the repo isn't available anymore (eg: > reboot for some reason) and takes back where it was when the repo is > back online. Here, the disks are connected via iSCSI. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From bkborg at kirk.de Wed May 10 11:40:24 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Wed, 10 May 2023 17:40:24 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: <76b14203-d2ec-7972-4829-f6aca06a7670@waldmann-edv.de> References: <76b14203-d2ec-7972-4829-f6aca06a7670@waldmann-edv.de> Message-ID: <0dd9978d-19ab-50af-134d-6447f760d9c3@kirk.de> Hello, Am 10.05.23 um 16:41 schrieb Thomas Waldmann: > Wow, that took long. Just 11.6 TB of compressed and deduplicated data (source data is about 17.2 TB, ratio ~68%). Disks (both, productive and backup disks) are mounted via 10GB ethernet line. It became slower day by day, and at he end if the time borg runs was too short (less than 3 hours continuously) the amount of backed up data was even negative. > Interrupted backup (connection issue, transport endpoint not connected): > just start borg create again and again in the same way. It won't > transmit the data again which it already has in the repo. For this, a > "borg check" is not needed. But what about the lock and the error messages? Should both disappear after some borg create runs? > I/O Error: fs or storage device malfunctioning, if you have access, run > a SMART long test on the disks and check the SMART status/logs (smartctl). It is on a Synology box. I just started scrubbing ("Datenbereinigung" in german) and a S.M.A.R.T. long test. Additionally, it offers an IronWolf Health check. Usually, scrubbing runs every 6 months, S.M.A.R.T. and IronWolf Health monthly. I'll quote if any errors are reported. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From tschoening at am-soft.de Wed May 10 11:46:07 2023 From: tschoening at am-soft.de (=?utf-8?Q?Thorsten_Sch=C3=B6ning?=) Date: Wed, 10 May 2023 17:46:07 +0200 Subject: [Borgbackup] S3 support? In-Reply-To: References: Message-ID: <244749127.20230510174607@am-soft.de> Guten Tag Evelyn Pereira Souza via Borgbackup, am Mittwoch, 10. Mai 2023 um 16:34 schrieben Sie: > It works great, but I want an offsite upload to Amazon S3 and have > not found anything regarding S3 support. I used to use s3cmd and gpg for AWS S3. BorgMatic supports backing up the same data to multiple different destinations, which is incremental and encrypted OOB because of Borg doing things. If you are able to mount your remote storage, it should be easy to use it: https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/ https://www.nakivo.com/blog/mount-amazon-s3-as-a-drive-how-to-guide/ Mit freundlichen Gr??en Thorsten Sch?ning -- AM-SoFT IT-Service - Bitstore Hameln GmbH Mitglied der Bitstore Gruppe - Ihr Full-Service-Dienstleister f?r IT und TK E-Mail: Thorsten.Schoening at AM-SoFT.de Web: http://www.AM-SoFT.de/ Tel: +49 5151- 9468- 0 Tel: +49 5151- 9468-55 Mobil: +49 178-8 9468-04 AM-SoFT IT-Service - Bitstore Hameln GmbH, Brandenburger Str. 7c, 31789 Hameln AG Hannover HRB 221853 - Gesch?ftsf?hrer: Janine Galonska F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From lazyvirus at gmx.com Wed May 10 12:26:40 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Wed, 10 May 2023 18:26:40 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: References: <20230510162740.668c7cf2@msi.defcon1.lan> Message-ID: <20230510182640.53215da5@msi.defcon1.lan> On Wed, 10 May 2023 16:51:40 +0200 Boris Kirkorowicz wrote: Hi, > >>> "/usr/lib64/python3.10/site-packages/borg/helpers/fs.py", line > >>> 199, in secure_erase with open(path, 'r+b') as fd: OSError: > >>> [Errno 5] Input/output error: '/mnt/rs1219a/usrlocal/config.old' > > ^^^^^^^^^^ > > Apparently, you have a problem on the above file, very possibly a > > bad sector which renders it unreadable, this might be bad for the > > rest of the repo. > > that seems not to be the case, since I can read it using cat and edit > it using vi. The error remains the same. The text file contains some > info about the repo: > > [repository] > > version = 1 > > segments_per_dir = 1000 > > max_segment_size = 524288000 > > append_only = 0 > > storage_quota = 0 > > additional_free_space = 0 > > id = > > 180ba330d0a627f2259f0ec4132fef7a45d135cf774b6c41ec75755744d5bdf9 As I told you, the FS can say it can't read the file while you can still manually copy it - I has this case 3 days ago with /boot/intrd? declared unreadable, but when I manually copied it, there was no problem (I rename the original one BAS_SECTOR and my copy the original name and that was it, but it has no importance if it dies fast, as this HD is 10 yo). > Could the error be that e.g. the id string is wrong, or something > similar? I don't think so (but I'm not sure, so Thomas may have the last word about that), if it was that, the logic says it would complain another way (such as: 'id' seems to be bad, did you cut the blue wire???) > > This is a problem as it could have two origin: a random bad sector > > on a big disk, which needs to be fixed before removing the file to > > avoid to hit on a BB file in the future, or it can be your disk > > dying. > > The disks are brand new, initialized using ext4 just before running > borg. Yeah, but HOW did you "initialized" them? Did you: mkfs.ext4 -cc ? ? Which formats disk(s) while performing a multi-patterns write/read (at least 4 IIRC) in order to reallocate/mark bad sectors if found. If disks have evolved, there is one of their spec that did not: the BER (Binary Error Rate), it is at the same level it was with 40/80 GB disks, so the chance you find bad sectors on big disks is very high. ZE problem is, it is veeery looong, as it: * writes the whole partition with the same pattern, * reads the whole partition to see if there is/are error(s), and loops on to the next pattern. IIRC, the last disk I format with this option was a 3TB 7500RPM which took something like 38 hours to achieve this kind of formatting? > > So, fix the bad sector, copy the repo on another disk (where you'll > > run the BB check) and do what's needed for the original disk (smart > > long test and formatting in ext4 with destructive patterns, which > > can take a looong time if the disk is big). > > AFAICS it applies to all repos. The backup took about four months to > complete, so formatting would not be that significant. So I hope that > the cause is something different and it could be repaired by another > method. It's your data, thus your ass Cochise ;-p) To be clear, if it was for myself, home data I don't really care as the important things have 2 different BB repos separated from the rest, but if it was professional data, I certainly wouldn't leave it this way, too much risks for a breakdown. Also note that if it is pro, there is a big bottleneck somewhere, seeing the time the 1st backup took, so the choice initially made probably may not be the best. Jean-Yves From panayotis at panayotis.com Wed May 10 12:31:56 2023 From: panayotis at panayotis.com (Panayotis Katsaloulis) Date: Wed, 10 May 2023 19:31:56 +0300 Subject: [Borgbackup] S3 support? In-Reply-To: <244749127.20230510174607@am-soft.de> References: <244749127.20230510174607@am-soft.de> Message-ID: <3e3ba70e-26cf-533a-4eb1-62cc57423c67@panayotis.com> For me, when mounting remote storage, I always had problems. See here: https://github.com/borgbackup/borg/issues/822 https://github.com/rclone/rclone/issues/3641 and other links, including my own (bad) experience. If you can have a remote borg server, it would be fine. If not, maybe search for alternatives. I, myself, for remote non-ssh related backups switched to another project. On 5/10/23 18:46, Thorsten Sch?ning wrote: >> It works great, but I want an offsite upload to Amazon S3 and have >> not found anything regarding S3 support. I used to use s3cmd and gpg for AWS S3. > BorgMatic supports backing up the same data to multiple different > destinations, which is incremental and encrypted OOB because of Borg > doing things. If you are able to mount your remote storage, it should > be easy to use it: > > https://torsion.org/borgmatic/docs/how-to/make-backups-redundant/ > https://www.nakivo.com/blog/mount-amazon-s3-as-a-drive-how-to-guide/ > > Mit freundlichen Gr??en > > Thorsten Sch?ning > From ContainerPi at disroot.org Wed May 10 13:06:20 2023 From: ContainerPi at disroot.org (Evelyn Pereira Souza) Date: Wed, 10 May 2023 19:06:20 +0200 Subject: [Borgbackup] S3 support? In-Reply-To: <3e3ba70e-26cf-533a-4eb1-62cc57423c67@panayotis.com> References: <244749127.20230510174607@am-soft.de> <3e3ba70e-26cf-533a-4eb1-62cc57423c67@panayotis.com> Message-ID: <9f43ebf6-e03c-40f5-e419-c2f2281a3c39@disroot.org> On 10.05.23 18:31, Panayotis Katsaloulis wrote: > For me, when mounting remote storage, I always had problems. > > See here: > https://github.com/borgbackup/borg/issues/822 > https://github.com/rclone/rclone/issues/3641 > and other links, including my own (bad) experience. > > If you can have a remote borg server, it would be fine. If not, maybe > search for alternatives. > > I, myself, for remote non-ssh related backups switched to another project. Hi Panayotis you switched to https://restic.net? kind regards Evelyn -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x61776FA8E38403FB.asc Type: application/pgp-keys Size: 2448 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 665 bytes Desc: OpenPGP digital signature URL: From panayotis at panayotis.com Wed May 10 13:10:37 2023 From: panayotis at panayotis.com (Panayotis Katsaloulis) Date: Wed, 10 May 2023 20:10:37 +0300 Subject: [Borgbackup] S3 support? In-Reply-To: <9f43ebf6-e03c-40f5-e419-c2f2281a3c39@disroot.org> References: <244749127.20230510174607@am-soft.de> <3e3ba70e-26cf-533a-4eb1-62cc57423c67@panayotis.com> <9f43ebf6-e03c-40f5-e419-c2f2281a3c39@disroot.org> Message-ID: <1d0022a1-da7b-359f-ba23-41beb3080a0c@panayotis.com> Yes, that's what I did. But for "local" and "ssh" backups, I still use borg. I see it as a way to reduce the danger, if something goes wrong. Different technologies, different applications, different media etc etc On 5/10/23 20:06, Evelyn Pereira Souza via Borgbackup wrote: > On 10.05.23 18:31, Panayotis Katsaloulis wrote: >> For me, when mounting remote storage, I always had problems. >> >> See here: >> https://github.com/borgbackup/borg/issues/822 >> https://github.com/rclone/rclone/issues/3641 >> and other links, including my own (bad) experience. >> >> If you can have a remote borg server, it would be fine. If not, maybe >> search for alternatives. >> >> I, myself, for remote non-ssh related backups switched to another >> project. > > Hi Panayotis > > you switched to https://restic.net? > > kind regards > Evelyn > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From ContainerPi at disroot.org Wed May 10 13:13:50 2023 From: ContainerPi at disroot.org (Evelyn Pereira Souza) Date: Wed, 10 May 2023 19:13:50 +0200 Subject: [Borgbackup] S3 support? In-Reply-To: <1d0022a1-da7b-359f-ba23-41beb3080a0c@panayotis.com> References: <244749127.20230510174607@am-soft.de> <3e3ba70e-26cf-533a-4eb1-62cc57423c67@panayotis.com> <9f43ebf6-e03c-40f5-e419-c2f2281a3c39@disroot.org> <1d0022a1-da7b-359f-ba23-41beb3080a0c@panayotis.com> Message-ID: <5e74a640-a7bc-9b49-7b80-364919eba381@disroot.org> On 10.05.23 19:10, Panayotis Katsaloulis wrote: > Yes, that's what I did. > > But for "local" and "ssh" backups, I still use borg. > > I see it as a way to reduce the danger, if something goes wrong. > Different technologies, different applications, different media etc etc Very good point. I have also thought about something similar. However, I have to learn so 2x tools and test the restore. -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_0x61776FA8E38403FB.asc Type: application/pgp-keys Size: 2448 bytes Desc: OpenPGP public key URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenPGP_signature Type: application/pgp-signature Size: 665 bytes Desc: OpenPGP digital signature URL: From tw at waldmann-edv.de Wed May 10 15:45:00 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 10 May 2023 21:45:00 +0200 Subject: [Borgbackup] recommended check option In-Reply-To: <0dd9978d-19ab-50af-134d-6447f760d9c3@kirk.de> References: <76b14203-d2ec-7972-4829-f6aca06a7670@waldmann-edv.de> <0dd9978d-19ab-50af-134d-6447f760d9c3@kirk.de> Message-ID: > Just 11.6 TB of compressed and deduplicated data (source data is about > 17.2 TB, ratio ~68%). Disks (both, productive and backup disks) are > mounted via 10GB ethernet line. It became slower day by day, I don't see a reason why borg should get (significantly) slower over time as long as it does not run out of resources (like RAM) - besides the startup and shutdown overhead (of course, loading and saving big repo index / chunks index and files cache takes more time than smaller ones). > and at he > end if the time borg runs was too short (less than 3 hours continuously) > the amount of backed up data was even negative. How can a data amount be negative? >> Interrupted backup (connection issue, transport endpoint not connected): >> just start borg create again and again in the same way. It won't >> transmit the data again which it already has in the repo. For this, a >> "borg check" is not needed. > > But what about the lock and the error messages? Should both disappear > after some borg create runs? If there is a lock in the repo, you'ld need to follow to "borg break-lock" docs. But borg usually avoids leaving back locks and also can detect and auto-remove stale locks in some circumstances (== if it can be sure). Running a rather recent borg version might help a bit with this, esp. if borg gets killed via some signal and the repo is via ssh:. >> I/O Error: fs or storage device malfunctioning, if you have access, run >> a SMART long test on the disks and check the SMART status/logs >> (smartctl). > > It is on a Synology box. I just started scrubbing ("Datenbereinigung" in > german) and a S.M.A.R.T. long test. Additionally, it offers an IronWolf > Health check. Usually, scrubbing runs every 6 months, S.M.A.R.T. and > IronWolf Health monthly. I'll quote if any errors are reported. OK, hopefully that will find the root cause of the IOError. From ndbecker2 at gmail.com Wed May 10 18:50:12 2023 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 10 May 2023 18:50:12 -0400 Subject: [Borgbackup] S3 support? In-Reply-To: <5e74a640-a7bc-9b49-7b80-364919eba381@disroot.org> References: <244749127.20230510174607@am-soft.de> <3e3ba70e-26cf-533a-4eb1-62cc57423c67@panayotis.com> <9f43ebf6-e03c-40f5-e419-c2f2281a3c39@disroot.org> <1d0022a1-da7b-359f-ba23-41beb3080a0c@panayotis.com> <5e74a640-a7bc-9b49-7b80-364919eba381@disroot.org> Message-ID: On Wed, May 10, 2023 at 1:14?PM Evelyn Pereira Souza via Borgbackup < borgbackup at python.org> wrote: > On 10.05.23 19:10, Panayotis Katsaloulis wrote: > > Yes, that's what I did. > > > > But for "local" and "ssh" backups, I still use borg. > > > > I see it as a way to reduce the danger, if something goes wrong. > > Different technologies, different applications, different media etc etc > > Very good point. > > I have also thought about something similar. However, I have to learn so > 2x tools and test the restore. > _______________________________________________ > I've also used 2 step, borg backup to my own machine (local or ssh), followed by rclone to save a remote copy -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkborg at kirk.de Sat May 13 06:53:02 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Sat, 13 May 2023 12:53:02 +0200 Subject: [Borgbackup] file changed while we backed it up Message-ID: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> Hi, last night I got two files reported with "file changed while we backed it up". Is there an option to get reported file date & size, rwx, owner:group, or whatever is meant? (The reported files are video files from last year that should not have changed.) What happens if this occurs: is the old file backed up, the new one, something between, or nothing? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From tw at waldmann-edv.de Sat May 13 07:59:19 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 13 May 2023 13:59:19 +0200 Subject: [Borgbackup] file changed while we backed it up In-Reply-To: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> References: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> Message-ID: <6eb204c3-d14b-3179-36c6-edf5fb6151d8@waldmann-edv.de> > last night I got two files reported with "file changed while we backed > it up". Is there an option to get reported file date & size, rwx, > owner:group, or whatever is meant? https://github.com/borgbackup/borg/blob/1.2.4/src/borg/archive.py#L1460 It detects changes in the ctime ("inode change time") timestamp of the file. > (The reported files are video files > from last year that should not have changed.) That's strange. Any cronjobs that do chown/chmod or ACL changes? Any search indexer or other tool modifying xattrs? > What happens if this occurs: is the old file backed up, the new one, > something between, or nothing? Something in between. How serious that is of course depends on the specific change (could be metadata, could be content data). borg 1.x just backs up whatever it gets from the filesystem. borg2 will do some retries in such a case. From tw at waldmann-edv.de Sat May 13 08:17:11 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 13 May 2023 14:17:11 +0200 Subject: [Borgbackup] file changed while we backed it up In-Reply-To: <6eb204c3-d14b-3179-36c6-edf5fb6151d8@waldmann-edv.de> References: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> <6eb204c3-d14b-3179-36c6-edf5fb6151d8@waldmann-edv.de> Message-ID: <79a52576-ce34-91b9-bbfc-8bee2bee0306@waldmann-edv.de> While thinking about it, I opened this issue with some ideas: https://github.com/borgbackup/borg/issues/7558 From lazyvirus at gmx.com Sat May 13 09:07:47 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Sat, 13 May 2023 15:07:47 +0200 Subject: [Borgbackup] file changed while we backed it up In-Reply-To: <79a52576-ce34-91b9-bbfc-8bee2bee0306@waldmann-edv.de> References: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> <6eb204c3-d14b-3179-36c6-edf5fb6151d8@waldmann-edv.de> <79a52576-ce34-91b9-bbfc-8bee2bee0306@waldmann-edv.de> Message-ID: <20230513150747.06fcbdb6@msi.defcon1.lan> On Sat, 13 May 2023 14:17:11 +0200 Thomas Waldmann wrote: > While thinking about it, I opened this issue with some ideas: > > https://github.com/borgbackup/borg/issues/7558 I'm not sure it would bring something, as ctime is an all-in-one. As file(s) being modified during a backup are really a very few (very often files from /var/log, sometimes Firefox xxxxxx.sqlite in my own case), the thing that could be interesting would be to memorize them in an array and once the backup of other files is complete, retry to backup them (may be more than once, let's say 3 times with a random delay before retry, just in case). Jean-Yves From bkborg at kirk.de Sat May 13 13:08:49 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Sat, 13 May 2023 19:08:49 +0200 Subject: [Borgbackup] file changed while we backed it up In-Reply-To: <79a52576-ce34-91b9-bbfc-8bee2bee0306@waldmann-edv.de> References: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> <6eb204c3-d14b-3179-36c6-edf5fb6151d8@waldmann-edv.de> <79a52576-ce34-91b9-bbfc-8bee2bee0306@waldmann-edv.de> Message-ID: <7fe1e29a-19bb-ea2c-5a04-b56fbeb55e16@kirk.de> Hi, Am 13.05.23 um 14:17 schrieb Thomas Waldmann: > While thinking about it, I opened this issue with some ideas: > > https://github.com/borgbackup/borg/issues/7558 possible could be that some reading access had taken place, so the atime changed. Or could automatic scrubbing (scheduled by DSM every 6 months) be the cause? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From jimpop at domainmail.org Sat May 13 14:25:15 2023 From: jimpop at domainmail.org (Jim Popovitch) Date: Sat, 13 May 2023 14:25:15 -0400 Subject: [Borgbackup] delete and prune use consume gigs of data Message-ID: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> Why does delete and prune seem to consume as much (or more) data as a create does, especially when the prune is done from a different location than where the create is initiated? I tried to delete some old server backups, using a session+key on my laptop, and the stats say my laptop xfer'ed in and out 1Mbs for 3 hours (that's a lot of transferred data for a delete). tia, -Jim P. From jimpop at domainmail.org Sat May 13 15:14:28 2023 From: jimpop at domainmail.org (Jim Popovitch) Date: Sat, 13 May 2023 15:14:28 -0400 Subject: [Borgbackup] delete and prune use consume gigs of data In-Reply-To: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> References: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> Message-ID: <4f9cb5933f56c85d612cf8420324b68d546c3901.camel@domainmail.org> On Sat, 2023-05-13 at 14:25 -0400, Jim Popovitch via Borgbackup wrote: > Why does delete and prune seem to consume as much (or more) data as a > create does, especially when the prune is done from a different location > than where the create is initiated? I tried to delete some old server > backups, using a session+key on my laptop, and the stats say my laptop > xfer'ed in and out 1Mbs for 3 hours (that's a lot of transferred data > for a delete). > Further insight reveals a new 1.5G .cache/borg folder. So I guess the delete needed to build a cache in order to know what to delete since the backups are encrypted? -Jim P. From TSchoening at am-soft.de Thu May 18 09:20:04 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Thu, 18 May 2023 13:20:04 +0000 Subject: [Borgbackup] file changed while we backed it up In-Reply-To: <20230513150747.06fcbdb6@msi.defcon1.lan> References: <3a4d4706-7a5f-5f74-b10c-6172694ee619@kirk.de> <6eb204c3-d14b-3179-36c6-edf5fb6151d8@waldmann-edv.de> <79a52576-ce34-91b9-bbfc-8bee2bee0306@waldmann-edv.de>, <20230513150747.06fcbdb6@msi.defcon1.lan> Message-ID: <1b9d584be548431c86e5b956e2174ddd@am-soft.de> _______________________________ F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz _____ Von: Borgbackup im Auftrag von Bzzzz Gesendet: Samstag, 13. Mai 2023 15:07 An: borgbackup at python.org Betreff: Re: [Borgbackup] file changed while we backed it up > [...]the thing that could be interesting would be to memorize them in > an array and once the backup of other files is complete, retry to backup > them (may be more than once, let's say 3 times with a random delay > before retry, just in case). In my opinion, Borg shouldn't invest too much time instead, as the only reliable way to backup files in use are concepts of snapshots anyway. Modern filesystems provide these any for any other use-case there isn't any guarantee at all anyway. From tw at waldmann-edv.de Thu May 18 12:39:30 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 18 May 2023 18:39:30 +0200 Subject: [Borgbackup] delete and prune use consume gigs of data In-Reply-To: <4f9cb5933f56c85d612cf8420324b68d546c3901.camel@domainmail.org> References: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> <4f9cb5933f56c85d612cf8420324b68d546c3901.camel@domainmail.org> Message-ID: <20e9bf07-f7c1-f7e4-aca8-8b0f0207c498@waldmann-edv.de> On 13.05.23 21:14, Jim Popovitch via Borgbackup wrote: > On Sat, 2023-05-13 at 14:25 -0400, Jim Popovitch via Borgbackup wrote: >> Why does delete and prune seem to consume as much (or more) data as a >> create does, especially when the prune is done from a different location >> than where the create is initiated? I tried to delete some old server >> backups, using a session+key on my laptop, and the stats say my laptop >> xfer'ed in and out 1Mbs for 3 hours (that's a lot of transferred data >> for a delete). >> > > Further insight reveals a new 1.5G .cache/borg folder. If that folder wasn't there before, borg had to do a re-sync of the chunks index. This is needed for prune/delete operations because it contains the refcounts for each chunk. Once the resync is done (and while it stays in sync), it will work quicker and with much less traffic. From jimpop at domainmail.org Thu May 18 13:25:59 2023 From: jimpop at domainmail.org (Jim Popovitch) Date: Thu, 18 May 2023 13:25:59 -0400 Subject: [Borgbackup] delete and prune use consume gigs of data In-Reply-To: <20e9bf07-f7c1-f7e4-aca8-8b0f0207c498@waldmann-edv.de> References: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> <4f9cb5933f56c85d612cf8420324b68d546c3901.camel@domainmail.org> <20e9bf07-f7c1-f7e4-aca8-8b0f0207c498@waldmann-edv.de> Message-ID: <1c31074051e3b0efe8d0bdfb2e9a269337bbb965.camel@domainmail.org> On Thu, 2023-05-18 at 18:39 +0200, Thomas Waldmann wrote: > > On 13.05.23 21:14, Jim Popovitch via Borgbackup wrote: > > On Sat, 2023-05-13 at 14:25 -0400, Jim Popovitch via Borgbackup wrote: > > > Why does delete and prune seem to consume as much (or more) data as a > > > create does, especially when the prune is done from a different location > > > than where the create is initiated? I tried to delete some old server > > > backups, using a session+key on my laptop, and the stats say my laptop > > > xfer'ed in and out 1Mbs for 3 hours (that's a lot of transferred data > > > for a delete). > > > > > > > Further insight reveals a new 1.5G .cache/borg folder. > > If that folder wasn't there before, borg had to do a re-sync of the > chunks index. This is needed for prune/delete operations because it > contains the refcounts for each chunk. > > Once the resync is done (and while it stays in sync), it will work > quicker and with much less traffic. Thanks for the explanation, indeed that folder was not there before. The backups are done from servers directly to remote storage, and I was doing the pruning from my laptop. Now that the servers have produced more backups (to the remote storage), should I expect to see ~/.cache/borg grow larger on my laptop the next time I prune? -Jim P. From tw at waldmann-edv.de Fri May 19 12:17:10 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Fri, 19 May 2023 18:17:10 +0200 Subject: [Borgbackup] delete and prune use consume gigs of data In-Reply-To: <1c31074051e3b0efe8d0bdfb2e9a269337bbb965.camel@domainmail.org> References: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> <4f9cb5933f56c85d612cf8420324b68d546c3901.camel@domainmail.org> <20e9bf07-f7c1-f7e4-aca8-8b0f0207c498@waldmann-edv.de> <1c31074051e3b0efe8d0bdfb2e9a269337bbb965.camel@domainmail.org> Message-ID: <02e8203c-35a8-a51f-f0c7-01af00801d0d@waldmann-edv.de> >> If that folder wasn't there before, borg had to do a re-sync of the >> chunks index. This is needed for prune/delete operations because it >> contains the refcounts for each chunk. >> >> Once the resync is done (and while it stays in sync), it will work >> quicker and with much less traffic. > > Thanks for the explanation, indeed that folder was not there before. The > backups are done from servers directly to remote storage, and I was > doing the pruning from my laptop. > Now that the servers have produced > more backups (to the remote storage), should I expect to see > ~/.cache/borg grow larger on my laptop the next time I prune? Yes, as soon as you do more backups from another machine to the repo, it will need to resync the chunks cache again and for doing that, it will fetch metadata of all new archives, build corresponding per-archive chunks indexes in the cache and compute a new overall chunks index from these cached per-archive chunks indexes. From jimpop at domainmail.org Fri May 19 12:23:41 2023 From: jimpop at domainmail.org (Jim Popovitch) Date: Fri, 19 May 2023 12:23:41 -0400 Subject: [Borgbackup] delete and prune use consume gigs of data In-Reply-To: <02e8203c-35a8-a51f-f0c7-01af00801d0d@waldmann-edv.de> References: <2450a3604dea88152781830e285ce1ebb0c40a03.camel@domainmail.org> <4f9cb5933f56c85d612cf8420324b68d546c3901.camel@domainmail.org> <20e9bf07-f7c1-f7e4-aca8-8b0f0207c498@waldmann-edv.de> <1c31074051e3b0efe8d0bdfb2e9a269337bbb965.camel@domainmail.org> <02e8203c-35a8-a51f-f0c7-01af00801d0d@waldmann-edv.de> Message-ID: <2e8af9c64e701e5bdca13e2d583bdbf9df803fbc.camel@domainmail.org> On Fri, 2023-05-19 at 18:17 +0200, Thomas Waldmann wrote: > > > If that folder wasn't there before, borg had to do a re-sync of > > > the chunks index. This is needed for prune/delete operations > > > because it contains the refcounts for each chunk. > > > > > > Once the resync is done (and while it stays in sync), it will > > > work quicker and with much less traffic. > > > > Thanks for the explanation, indeed that folder was not there > > before. The backups are done from servers directly to remote > > storage, and I was doing the pruning from my laptop. > > > Now that the servers have produced more backups (to the remote > > storage), should I expect to see ~/.cache/borg grow larger on my > > laptop the next time I prune? > > Yes, as soon as you do more backups from another machine to the > repo, it will need to resync the chunks cache again and for doing > that, it will fetch metadata of all new archives, build > corresponding per-archive chunks indexes in the cache and compute a > new overall chunks index from these cached per-archive chunks > indexes. > Got it, Thanks!! -Jim P. From bkborg at kirk.de Sun May 21 13:24:54 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Sun, 21 May 2023 19:24:54 +0200 Subject: [Borgbackup] leftover repos Message-ID: Hi, I tried some experiments with dummy directories to find out the best compression method and level, and after that, the repos created should be removed. I quickly did this at file system level, simply by rm -rf /mnt/rs1219a/testrepo* Some of these directories weren't deleted because they aren't empty, but there were no files visible. So i tried borg delete, borg prune etc., but it did not work: repo not found or false parameter. After rebooting the target several times, I got rid of the directories, but ls -1 /mnt/borg (the borgfs mount) shows all of the testing repos: testrepo_zstd01 testrepo_zstd03 testrepo_zstd06 testrepo_zstd09 testrepo_zstd11 testrepo_zstd13 testrepo_zstd16 testrepo_zstd19 testrepo_zstd22 Now I wonder how to remove them. What do I have to do? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From lazyvirus at gmx.com Sun May 21 13:51:40 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Sun, 21 May 2023 19:51:40 +0200 Subject: [Borgbackup] leftover repos In-Reply-To: References: Message-ID: <20230521195140.4bd67ff7@msi.defcon1.lan> On Sun, 21 May 2023 19:24:54 +0200 Boris Kirkorowicz wrote: Hi, > I tried some experiments with dummy directories to find out the best > compression method and level, and after that, the repos created > should be removed. I quickly did this at file system level, simply by > > rm -rf /mnt/rs1219a/testrepo* > > Some of these directories weren't deleted because they aren't empty, > but there were no files visible. So i tried borg delete, borg prune > etc., but it did not work: repo not found or false parameter. > > After rebooting the target several times, I got rid of the > directories, but > > ls -1 /mnt/borg Let's recapitulate : * you removed DIR A * you find strange to find things into DIR B Am I the only one to see a tiny bit of discrepancy here ? > (the borgfs mount) shows all of the testing repos: > testrepo_zstd01 > testrepo_zstd03 > testrepo_zstd06 > testrepo_zstd09 > testrepo_zstd11 > testrepo_zstd13 > testrepo_zstd16 > testrepo_zstd19 > testrepo_zstd22 > > Now I wonder how to remove them. What do I have to do? eg.. rm -r /mnt/borg ? Jean-Yves From bkborg at kirk.de Sun May 21 14:04:45 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Sun, 21 May 2023 20:04:45 +0200 Subject: [Borgbackup] leftover repos In-Reply-To: <20230521195140.4bd67ff7@msi.defcon1.lan> References: <20230521195140.4bd67ff7@msi.defcon1.lan> Message-ID: <8e78cd4a-84bd-f8cb-4858-38423c16dbef@kirk.de> Hi, Am 21.05.23 um 19:51 schrieb Bzzzz: > Let's recapitulate : > * you removed DIR A > * you find strange to find things into DIR B > > Am I the only one to see a tiny bit of discrepancy here ? since my understanding was that borgfs gets it's data from the repos, I assumed that there might be another way for borgfs than just the directories that don't exist any more, eg. a borg database or something like that. If so, it might not be the clean way just to remove the directories and leave some dead stuff there. So I thought it was a good idea to ask before than lugging something around. If you remember, I have some performance problems and do not need any ballast. But if it is safe just to delete these directories, I'll do that. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From lazyvirus at gmx.com Sun May 21 14:36:27 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Sun, 21 May 2023 20:36:27 +0200 Subject: [Borgbackup] leftover repos In-Reply-To: <8e78cd4a-84bd-f8cb-4858-38423c16dbef@kirk.de> References: <20230521195140.4bd67ff7@msi.defcon1.lan> <8e78cd4a-84bd-f8cb-4858-38423c16dbef@kirk.de> Message-ID: <20230521203627.3961b942@msi.defcon1.lan> On Sun, 21 May 2023 20:04:45 +0200 Boris Kirkorowicz wrote: > Hi, > > Am 21.05.23 um 19:51 schrieb Bzzzz: > > Let's recapitulate : > > * you removed DIR A > > * you find strange to find things into DIR B > > > > Am I the only one to see a tiny bit of discrepancy here ? > > since my understanding was that borgfs gets it's data from the repos, Correct. > I assumed that there might be another way for borgfs than just the > directories that don't exist any more, eg. a borg database or > something like that. Wrong, there's not such thing in BB (everything's located in the repo). > If so, it might not be the clean way just to > remove the directories and leave some dead stuff there. So I thought > it was a good idea to ask before than lugging something around. Looks like you zapped the repo when it's content was still borgfs mounted, this is the only thing I see that could trigger this kind of situation. > If > you remember, I have some performance problems and do not need any > ballast. Normally, a simple borg umount /path/to/borgfs/mount should do the trick, but it might fail as there's nothing left of the repo - if it is the case, simply reboot, that should cure the whole thing. > But if it is safe just to delete these directories, I'll do that. I missed the borgfs part at first - you can't delete anything, as the borgfs mount is _always_ done R/O. Jean-Yves From TSchoening at am-soft.de Mon May 22 09:11:10 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Mon, 22 May 2023 13:11:10 +0000 Subject: [Borgbackup] leftover repos In-Reply-To: References: Message-ID: <4ba040001eb14b6fb77560f034248620@am-soft.de> Borgbackup im Auftrag von Boris Kirkorowicz wrote on Sonntag, 21. Mai 2023 19:24: > (the borgfs mount) shows all of the testing repos: > testrepo_zstd01 Looks to me like you are wrongly mixing repos with archives and their dirs in those repos mounted by borgfs. You can't delete dirs within archives mounted by borgfs and your can't delete repos still mounted by borgfs. So stop mounting using borgfs and afterwards it should be easily to delete the repos. -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From bkborg at kirk.de Thu May 25 08:06:15 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Thu, 25 May 2023 14:06:15 +0200 Subject: [Borgbackup] borg delete failed Message-ID: Hi, connected via ssh to sfo2205, I tried to delete a Repo using borg delete. It's answer: > sfo2205:~ # borg delete /mnt/rs1219a/Boris > You requested to completely DELETE the following repository *including* 105 archives it contains: > ------------------------------------------------------------------------------ > Repository ID: 49384c5e96328d2133e8f206cdc2b6107889b16eaeaa702d9db8e63d9b936120 > Location: /mnt/rs1219a/Boris > ------------------------------------------------------------------------------ > Type 'YES' if you understand this and want to continue: YES > Local Exception > Traceback (most recent call last): > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5213, in main > exit_code = archiver.run(args) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 5144, in run > return set_ec(func(args)) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 183, in wrapper > return method(self, args, repository=repository, **kwargs) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 1200, in do_delete > return self._delete_repository(args, repository) > File "/usr/lib64/python3.10/site-packages/borg/archiver.py", line 1334, in _delete_repository > repository.destroy() > File "/usr/lib64/python3.10/site-packages/borg/repository.py", line 380, in destroy > shutil.rmtree(self.path) > File "/usr/lib64/python3.10/shutil.py", line 725, in rmtree > _rmtree_safe_fd(fd, path, onerror) > File "/usr/lib64/python3.10/shutil.py", line 658, in _rmtree_safe_fd > _rmtree_safe_fd(dirfd, fullname, onerror) > File "/usr/lib64/python3.10/shutil.py", line 664, in _rmtree_safe_fd > onerror(os.rmdir, fullname, sys.exc_info()) > File "/usr/lib64/python3.10/shutil.py", line 662, in _rmtree_safe_fd > os.rmdir(entry.name, dir_fd=topfd) > OSError: [Errno 39] Directory not empty: '20' > > Platform: Linux sfo2205 6.1.7-1-default #1 SMP PREEMPT_DYNAMIC Wed Jan 18 11:12:34 UTC 2023 (872045c) x86_64 > Linux: Unknown Linux > Borg: 1.2.4 Python: CPython 3.10.10 msgpack: 1.0.4 fuse: pyfuse3 3.2.2 [pyfuse3,llfuse] > PID: 13594 CWD: /root > sys.argv: ['/usr/bin/borg', 'delete', '/mnt/rs1219a/Boris'] > SSH_ORIGINAL_COMMAND: None I am not sure what that means. There were a "data" subdir, containing further subdirs. By now, I solved this by > rm -r /mnt/rs1219a/Boris but I like to understand what went wrong with borg delete. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From tw at waldmann-edv.de Sat May 27 05:01:44 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 27 May 2023 11:01:44 +0200 Subject: [Borgbackup] borg delete failed In-Reply-To: References: Message-ID: <505d1cdc-e469-3e76-165f-d9ae75028f30@waldmann-edv.de> >> ? File "/usr/lib64/python3.10/site-packages/borg/repository.py", line >> 380, in destroy >> ??? shutil.rmtree(self.path) That call should completely remove the directory and all stuff inside. >> OSError: [Errno 39] Directory not empty: '20' But it failed to do so, because there was something inside a directory when it tried to rmdir that directory. It usually does the rmdir **after** removing all stuff inside the directory (files and subdirs), so that is unexpected. Possible explanations: - there was acticity in that directory in parallel (should not be borg, because borg delete locks the repository). did you use borg break-lock? - filesystem malfunctioning - bug in shutil.rmtree > I am not sure what that means. There were a "data" subdir, containing > further subdirs. By now, I solved this by > >> rm -r /mnt/rs1219a/Boris Yeah, but that should do a quite similar thing as shutil.rmtree does. Maybe it was successful because then there was no parallel activity or the fs behaved better that time. From bkborg at kirk.de Sun May 28 05:03:10 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Sun, 28 May 2023 11:03:10 +0200 Subject: [Borgbackup] file changed while we backed it up Message-ID: <6f671be0-4f02-bae6-71fd-d4cab6af49a0@kirk.de> Hi, during the last days, there are some messages like this: > borg create -v --stats --show-rc -C zstd /mnt/rs1219a/Boris::'{now:%Y-%m-%d_%H-%M}' /home/boris > Creating archive at "/mnt/rs1219a/Boris::2023-05-27_08-41" > /home/boris/Dokumente/Sicherungen/T560B/T560B-orig-4.img.gz: file changed while we backed it up > /home/boris/Dokumente/Sicherungen/T560B/T560B-orig.img.gz: file changed while we backed it up resulting in rc=1 I am pretty sure that these files didn't change since 2016, since they are backups of partition images that I only might need if I sell my old notebook some day. What does that message mean, what might cause it, how to avoid it? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From TSchoening at am-soft.de Mon May 29 06:42:42 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Mon, 29 May 2023 10:42:42 +0000 Subject: [Borgbackup] file changed while we backed it up In-Reply-To: <6f671be0-4f02-bae6-71fd-d4cab6af49a0@kirk.de> References: <6f671be0-4f02-bae6-71fd-d4cab6af49a0@kirk.de> Message-ID: <8084cb08bfa648919cabeb815a5446a7@am-soft.de> Borgbackup im Auftrag von Boris Kirkorowicz wrote at Gesendet: Sonntag, 28. Mai 2023 11:03: > /home/boris/Dokumente/Sicherungen/T560B/T560B-orig-4.img.gz: file changed while we backed it up > I am pretty sure that these files didn't change since 2016[...] There was an issue created recently describing the context of that message, it's not only about really changing blocks within that file, but might be about changed permissions or stuff as well. It might be the result of some concurrent process doing things to the files the same time Borg tries to back them up. Especially because you have written about multiple problems in the last weeks which read like concurrency issues somewhere, I would start looking at those topics first if I was you. https://github.com/borgbackup/borg/issues/7558#issuecomment-1546652883 > [...]how to avoid it? You need to create some sort of read-only snapshot of the filesystem, mount that and backup the files from that. There is no other realiable way to prevent (accidently) changing files during backups. With tools like BorgMatic and its hooks this is somewhat easy to achive: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/ -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From ldl08 at gmx.net Tue May 30 08:50:36 2023 From: ldl08 at gmx.net (ldl08 at gmx.net) Date: Tue, 30 May 2023 14:50:36 +0200 Subject: [Borgbackup] Permissions -- I erroneously ran Borg as root; how can I go back to user usage Message-ID: Dear Borg friends, https://borgbackup.readthedocs.io/en/stable/quickstart.html has a clear warning: "To avoid permissions issues (in your borg repository or borg cache), always access the repository using the same user account. If you want to backup files of other users or the operating system, running borg as root likely will be required (otherwise you?ld get Permission denied errors). If you only back up your own files, you neither need nor want to run borg as root, just run it as your normal user." Well, I was trying some stuff out, and as it happens I ran Borg as root. Now I get an error message: PermissionError: [Errno 13] Permission denied: '/path/to/my/repository' I can continue to run it as root, but I'd like to go back to the old time. Is that possible, and if so, how? Thanks for your insights! David From aleksandr.kiselyov at gmail.com Tue May 30 10:18:28 2023 From: aleksandr.kiselyov at gmail.com (Alexander Kiselyov) Date: Tue, 30 May 2023 17:18:28 +0300 Subject: [Borgbackup] Permissions -- I erroneously ran Borg as root; how can I go back to user usage In-Reply-To: References: Message-ID: <13821a918195454b2543f2e32f7ba3435f6d3535.camel@gmail.com> It should be enough just to normally change owner as Borg shouldn't change file mode flags. If you have sudo configured, it can be done as follows: $ sudo chown -R $(id -u):$(id -g) /path/to/my/repository The commands in $() are evaluated as your regular user, -R means recursive operation. Hope that helps. Regards, Alexander On Tue, 2023-05-30 at 14:50 +0200, ldl08 at gmx.net wrote: > Dear Borg friends, > > https://borgbackup.readthedocs.io/en/stable/quickstart.html?has a > clear warning: > > "To avoid permissions issues (in your borg repository or borg cache), > always access the repository using the same user account. > > If you want to backup files of other users or the operating system, > running borg as root likely will be required (otherwise you?ld get > Permission denied errors). If you only back up your own files, you > neither need nor want to run borg as root, just run it as your normal > user." > > Well, I was trying some stuff out, and as it happens I ran Borg as > root. Now I get an error message: > > PermissionError: [Errno 13] Permission denied: > '/path/to/my/repository' > > I can continue to run it as root, but I'd like to go back to the old > time. Is that possible, and if so, how? > > Thanks for your insights! > > David > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup From info at rfechner.de Wed May 31 11:12:22 2023 From: info at rfechner.de (Ralf Fechner) Date: Wed, 31 May 2023 17:12:22 +0200 Subject: [Borgbackup] Crontab Message-ID: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> Sorry in advance for the deepl-English. I have set up Borg on MacOS X and it runs as it should. Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. Unfortunately the script is no longer executed correctly via the configured crontab, here it always says, Remote: Permission denied, please try again. But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? -- Mit freundlichen Gr??en Ralf Fechner Ende-zu-Ende-Verschl?sselung: PGP-Fingerprint: 48E9 23AE E27A BAB7 A68D 76F9 3483 D501 0AB9 9E0E -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From TSchoening at am-soft.de Wed May 31 13:07:46 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Wed, 31 May 2023 17:07:46 +0000 Subject: [Borgbackup] Crontab In-Reply-To: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> Message-ID: <423ad2e3cad1419a9f8d07aaeecef1f0@am-soft.de> Ralf Fechner via Borgbackup wrote at Mittwoch, 31. Mai 2023 17:12: > Now I have created a new SSH key to access the server. I no longer have this Key in > the .ssh directory, but I load this key into the ssh agend using KeepassXC. > Unfortunately the script is no longer executed correctly via the configured crontab, > here it always says, Remote: Permission denied, please try again. ".ssh directory" and "crontab" don't tell too much, you need to tell/check under which user the script is executed in the end. ".ssh" might have been properly available for that user in the past, but SSH agent might run for a different user. Additionally, SSH in your setup needs ot be told to use the agent actually, so it might make sense to provide your uses SSH arguments or configs. https://smallstep.com/blog/ssh-agent-explained/ -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From imperator at jedimail.de Wed May 31 13:26:59 2023 From: imperator at jedimail.de (Imperator) Date: Wed, 31 May 2023 19:26:59 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> Message-ID: <78dd0620-8c20-9ef8-2e48-44b43f4e31b6@jedimail.de> Hi Ralf, you could run bash -l -c "/path/to/your/script.sh" in your crontab to make sure that the script is executed just like you would do manually. Am 31.05.23 um 17:12 schrieb Ralf Fechner via Borgbackup: > But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? From bkborg at kirk.de Thu Jun 1 04:27:22 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Thu, 1 Jun 2023 10:27:22 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> Message-ID: <85d7bcc3-e02a-ee54-5077-ecc471a74140@kirk.de> Hi, make sure that you always give full paths, even if you don't need when invoked in a terminal session. This refers to crontab entries as well as scripts. Am 31.05.23 um 17:12 schrieb Ralf Fechner via Borgbackup: > Sorry in advance for the deepl-English. > > I have set up Borg on MacOS X and it runs as it should. > Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. > Unfortunately the script is no longer executed correctly via the configured crontab, here it always says, Remote: Permission denied, please try again. > But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? > > > -- > Mit freundlichen Gr??en > Ralf Fechner > > Ende-zu-Ende-Verschl?sselung: > PGP-Fingerprint: 48E9 23AE E27A BAB7 A68D 76F9 3483 D501 0AB9 9E0E > > > > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Mit freundlichem Gru? Best regards ? Kirkorowicz From info at rfechner.de Thu Jun 1 06:42:53 2023 From: info at rfechner.de (Ralf Fechner) Date: Thu, 1 Jun 2023 12:42:53 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <85d7bcc3-e02a-ee54-5077-ecc471a74140@kirk.de> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <85d7bcc3-e02a-ee54-5077-ecc471a74140@kirk.de> Message-ID: -- Mit freundlichen Gr??en Ralf Fechner > Am 01.06.2023 um 10:27 schrieb Boris Kirkorowicz : > > Hi, > make sure that you always give full paths, even if you don't need when invoked in a terminal session. This refers to crontab entries as well as scripts. > Unfortunately, the specification of the full path does not lead to the desired result, it remains the same, if I start the script manually, Borg works normally, if the script is started from the Crontab, the job is not executed. > > Am 31.05.23 um 17:12 schrieb Ralf Fechner via Borgbackup: >> Sorry in advance for the deepl-English. >> I have set up Borg on MacOS X and it runs as it should. >> Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. >> Unfortunately the script is no longer executed correctly via the configured crontab, here it always says, Remote: Permission denied, please try again. >> But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? >> -- >> Mit freundlichen Gr??en >> Ralf Fechner >> Ende-zu-Ende-Verschl?sselung: >> PGP-Fingerprint: 48E9 23AE E27A BAB7 A68D 76F9 3483 D501 0AB9 9E0E >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > -- > Mit freundlichem Gru? Best regards > Kirkorowicz > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From info at rfechner.de Thu Jun 1 06:52:52 2023 From: info at rfechner.de (Ralf Fechner) Date: Thu, 1 Jun 2023 12:52:52 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <423ad2e3cad1419a9f8d07aaeecef1f0@am-soft.de> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <423ad2e3cad1419a9f8d07aaeecef1f0@am-soft.de> Message-ID: <06AC52A3-EACC-4A32-B0DA-F5FBD655945B@rfechner.de> > Am 31.05.2023 um 19:07 schrieb AM-Soft ? Thorsten Sch?ning : > > Ralf Fechner via Borgbackup wrote at Mittwoch, 31. Mai 2023 17:12: > >> Now I have created a new SSH key to access the server. I no longer have this Key in >> the .ssh directory, but I load this key into the ssh agend using KeepassXC. >> Unfortunately the script is no longer executed correctly via the configured crontab, >> here it always says, Remote: Permission denied, please try again. > > ".ssh directory" and "crontab" don't tell too much, you need to tell/check under which > user the script is executed in the end. ".ssh" might have been properly available for > that user in the past, but SSH agent might run for a different user. Additionally, > SSH in your setup needs ot be told to use the agent actually, so it might make sense to > provide your uses SSH arguments or configs. > > https://smallstep.com/blog/ssh-agent-explained/ The script is executed under my user. The SSH agent is also running under my user. With ssh-add -l I can also see that the key is loaded correctly from KeepassXC to the agent. If I start the Borg script manually in the terminal as mentioned, it is also executed correctly. Only if the script is started via a crontab, the connection is rejected. > > -- > > F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. > > Mit freundlichen Gr??en, > > Thorsten Sch?ning > > > Telefon: +49 5151 9468-55 > Fax: +49 5151 9468-88 > E-Mail: TSchoening at am-soft.de > > AM-Soft IT-Service - Bitstore Hameln GmbH > Brandenburger Stra?e 7c > 31789 Hameln > > Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. > > This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. > > Hinweise zum Datenschutz: bitstore.group/datenschutz > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From info at rfechner.de Thu Jun 1 06:57:04 2023 From: info at rfechner.de (Ralf Fechner) Date: Thu, 1 Jun 2023 12:57:04 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <78dd0620-8c20-9ef8-2e48-44b43f4e31b6@jedimail.de> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <78dd0620-8c20-9ef8-2e48-44b43f4e31b6@jedimail.de> Message-ID: > Am 31.05.2023 um 19:26 schrieb Imperator : > > Hi Ralf, > > you could run > > bash -l -c "/path/to/your/script.sh" Unfortunately, the specifications have the same result. The problem remains the same. > > in your crontab to make sure that the script is executed just like you would do manually. > > Am 31.05.23 um 17:12 schrieb Ralf Fechner via Borgbackup: >> But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From bkborg at kirk.de Thu Jun 1 06:48:00 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Thu, 1 Jun 2023 12:48:00 +0200 Subject: [Borgbackup] Crontab In-Reply-To: References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <85d7bcc3-e02a-ee54-5077-ecc471a74140@kirk.de> Message-ID: Hi, maybe the environment is not the same. Setup a script that writes it to a logfile, and start this script once from command line and once from crontab to see the differences. If found, set the missing values in your script and run it from crontab. Am 01.06.23 um 12:42 schrieb Ralf Fechner via Borgbackup: > > > -- > Mit freundlichen Gr??en > Ralf Fechner > > > > >> Am 01.06.2023 um 10:27 schrieb Boris Kirkorowicz : >> >> Hi, >> make sure that you always give full paths, even if you don't need when invoked in a terminal session. This refers to crontab entries as well as scripts. >> > > > > Unfortunately, the specification of the full path does not lead to the desired result, it remains the same, if I start the script manually, Borg works normally, if the script is started from the Crontab, the job is not executed. > > > > > >> >> Am 31.05.23 um 17:12 schrieb Ralf Fechner via Borgbackup: >>> Sorry in advance for the deepl-English. >>> I have set up Borg on MacOS X and it runs as it should. >>> Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. >>> Unfortunately the script is no longer executed correctly via the configured crontab, here it always says, Remote: Permission denied, please try again. >>> But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? >>> -- >>> Mit freundlichen Gr??en >>> Ralf Fechner >>> Ende-zu-Ende-Verschl?sselung: >>> PGP-Fingerprint: 48E9 23AE E27A BAB7 A68D 76F9 3483 D501 0AB9 9E0E >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >> >> -- >> Mit freundlichem Gru? Best regards >> Kirkorowicz >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -- Mit freundlichem Gru? Best regards ? Kirkorowicz From info at rfechner.de Thu Jun 1 07:10:42 2023 From: info at rfechner.de (Ralf Fechner) Date: Thu, 1 Jun 2023 13:10:42 +0200 Subject: [Borgbackup] Crontab In-Reply-To: References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <85d7bcc3-e02a-ee54-5077-ecc471a74140@kirk.de> Message-ID: -- Mit freundlichen Gr??en Ralf Fechner Ende-zu-Ende-Verschl?sselung: PGP-Fingerprint: 48E9 23AE E27A BAB7 A68D 76F9 3483 D501 0AB9 9E0E > Am 01.06.2023 um 12:48 schrieb Boris Kirkorowicz : > > Hi, > maybe the environment is not the same. Setup a script that writes it to a logfile, and start this script once from command line and once from crontab to see the differences. If found, set the missing values in your script and run it from crontab. > The log file says when running above the crontab: Remote: Permission denied, please try again. Remote: Permission denied, please try again. Remote: server.com: Permission denied (publickey,password). Connection closed by remote host. Is borg working on the server? terminating with error status, rc 2 When run manually, the backup runs smoothly. > > > Am 01.06.23 um 12:42 schrieb Ralf Fechner via Borgbackup: >> -- >> Mit freundlichen Gr??en >> Ralf Fechner >>> Am 01.06.2023 um 10:27 schrieb Boris Kirkorowicz : >>> >>> Hi, >>> make sure that you always give full paths, even if you don't need when invoked in a terminal session. This refers to crontab entries as well as scripts. >>> >> Unfortunately, the specification of the full path does not lead to the desired result, it remains the same, if I start the script manually, Borg works normally, if the script is started from the Crontab, the job is not executed. >>> >>> Am 31.05.23 um 17:12 schrieb Ralf Fechner via Borgbackup: >>>> Sorry in advance for the deepl-English. >>>> I have set up Borg on MacOS X and it runs as it should. >>>> Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. >>>> Unfortunately the script is no longer executed correctly via the configured crontab, here it always says, Remote: Permission denied, please try again. >>>> But if I execute this crontab manually in the terminal, it works fine. Why does it not work when the crontab is executed automatically? >>>> -- >>>> Mit freundlichen Gr??en >>>> Ralf Fechner >>>> Ende-zu-Ende-Verschl?sselung: >>>> PGP-Fingerprint: 48E9 23AE E27A BAB7 A68D 76F9 3483 D501 0AB9 9E0E >>>> _______________________________________________ >>>> Borgbackup mailing list >>>> Borgbackup at python.org >>>> https://mail.python.org/mailman/listinfo/borgbackup >>> >>> -- >>> Mit freundlichem Gru? Best regards >>> Kirkorowicz >>> _______________________________________________ >>> Borgbackup mailing list >>> Borgbackup at python.org >>> https://mail.python.org/mailman/listinfo/borgbackup >> _______________________________________________ >> Borgbackup mailing list >> Borgbackup at python.org >> https://mail.python.org/mailman/listinfo/borgbackup > > -- > Mit freundlichem Gru? Best regards > Kirkorowicz > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From borgbackup at aluaces.fastmail.com Thu Jun 1 07:53:36 2023 From: borgbackup at aluaces.fastmail.com (Alberto Luaces) Date: Thu, 01 Jun 2023 13:53:36 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> (Ralf Fechner via Borgbackup's message of "Wed, 31 May 2023 17:12:22 +0200") References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> Message-ID: <87cz2fz9dr.fsf@eps142.cdf.udc.es> Ralf Fechner via Borgbackup writes: > Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. You can run ?ssh-add -L? from your cron script and inspect the logs. My bet is that the environment variables the ssh agent exports are not available when running the script through cron, thus ssh cannot reach the key. From info at rfechner.de Thu Jun 1 08:03:35 2023 From: info at rfechner.de (Ralf Fechner) Date: Thu, 1 Jun 2023 14:03:35 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <87cz2fz9dr.fsf@eps142.cdf.udc.es> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <87cz2fz9dr.fsf@eps142.cdf.udc.es> Message-ID: <7D690696-FBFE-4339-AD27-FB25F8993E30@rfechner.de> > Am 01.06.2023 um 13:53 schrieb Alberto Luaces : > > Ralf Fechner via Borgbackup writes: > >> Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. > > You can run ?ssh-add -L? from your cron script and inspect the logs. > > My bet is that the environment variables the ssh agent exports are not > available when running the script through cron, thus ssh cannot reach > the key. Ok, I have done that. Then comes the message: Could not open a connection to your authentication agent. This means that Crontab can not access the agent. Is it possible to change this? If yes how? > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From borgbackup at aluaces.fastmail.com Thu Jun 1 09:32:26 2023 From: borgbackup at aluaces.fastmail.com (Alberto Luaces) Date: Thu, 01 Jun 2023 15:32:26 +0200 Subject: [Borgbackup] Crontab In-Reply-To: <7D690696-FBFE-4339-AD27-FB25F8993E30@rfechner.de> (Ralf Fechner via Borgbackup's message of "Thu, 1 Jun 2023 14:03:35 +0200") References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <87cz2fz9dr.fsf@eps142.cdf.udc.es> <7D690696-FBFE-4339-AD27-FB25F8993E30@rfechner.de> Message-ID: <878rd3z4t1.fsf@eps142.cdf.udc.es> Ralf Fechner via Borgbackup writes: >> Am 01.06.2023 um 13:53 schrieb Alberto Luaces : >> >> Ralf Fechner via Borgbackup writes: >> >>> Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. >> >> You can run ?ssh-add -L? from your cron script and inspect the logs. >> >> My bet is that the environment variables the ssh agent exports are not >> available when running the script through cron, thus ssh cannot reach >> the key. > > Ok, I have done that. Then comes the message: > > Could not open a connection to your authentication agent. > > This means that Crontab can not access the agent. Is it possible to change this? If yes how? > You can copy the content of your $SSH_AUTH_SOCK environment variable in the cron job, but that is likely to change from time to time, so it is not a long-term solution. The usual procedure is to use a passwordless key, but restricting it to just running borg on the remote side, as shown in https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html#ssh-agent From tbr at mannynkapy.net Thu Jun 1 17:16:13 2023 From: tbr at mannynkapy.net (Tom Rushworth) Date: Thu, 1 Jun 2023 14:16:13 -0700 Subject: [Borgbackup] Crontab In-Reply-To: <878rd3z4t1.fsf@eps142.cdf.udc.es> References: <4AB78DEA-DE88-4D8E-AAA1-EA6D5B6EDEE7@rfechner.de> <87cz2fz9dr.fsf@eps142.cdf.udc.es> <7D690696-FBFE-4339-AD27-FB25F8993E30@rfechner.de> <878rd3z4t1.fsf@eps142.cdf.udc.es> Message-ID: Hi, On Thu, Jun 01, 2023 at 03:32:26PM +0200, Alberto Luaces wrote: > Ralf Fechner via Borgbackup writes: > > >> Am 01.06.2023 um 13:53 schrieb Alberto Luaces : > >> > >> Ralf Fechner via Borgbackup writes: > >> > >>> Now I have created a new SSH key to access the server. I no longer have this Key in the .ssh directory, but I load this key into the ssh agend using KeepassXC. > >> > >> You can run ?ssh-add -L? from your cron script and inspect the logs. > >> > >> My bet is that the environment variables the ssh agent exports are not > >> available when running the script through cron, thus ssh cannot reach > >> the key. > > > > Ok, I have done that. Then comes the message: > > > > Could not open a connection to your authentication agent. > > > > This means that Crontab can not access the agent. Is it possible to change this? If yes how? > > > > You can copy the content of your $SSH_AUTH_SOCK environment variable in > the cron job, but that is likely to change from time to time, so it is > not a long-term solution. You are better off starting an ssh-agent in your crontab script. Here is what I use at the start of a script for an unattended backup by user root on a FreeBSD machine: eval `/usr/bin/ssh-agent -s` /usr/bin/ssh-add /root/.ssh/private_key_for_backup > > The usual procedure is to use a passwordless key, but restricting it to > just running borg on the remote side, as shown in > https://borgbackup.readthedocs.io/en/stable/deployment/pull-backup.html#ssh-agent Right. The "private_key_for_backup" is just such a key. I use the ssh "user at host" notation so that there is no need to run as root on the remote host (backup server). The username I have on the remote host is "backup", so the borg command in the crontab script looks like: borg create backup at remote_host:remote_repo_name::archive_name dirs... The sshd config entry for the user "backup" on the remote host has been set up to restrict to just running borg (as mentioned above). I will probably have to change the '::' with the latest borg, but the stuff above works for now. > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup Cheers, -- Tom Rushworth From tw at waldmann-edv.de Mon Jun 12 04:48:20 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 12 Jun 2023 10:48:20 +0200 Subject: [Borgbackup] borg 2.0.0b6 released Message-ID: <9c6e3b7b-e5df-6342-ff6c-a26229bb24cc@waldmann-edv.de> Just released borg 2.0.0 beta6 - please help testing this and please read the change log! https://github.com/borgbackup/borg/releases/tag/2.0.0b6 #borgbackup #linux #freebsd #netbsd #openbsd #macOS From bkborg at kirk.de Wed Jun 14 01:55:30 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Wed, 14 Jun 2023 07:55:30 +0200 Subject: [Borgbackup] (too) many modified files Message-ID: Hi, looking for possible reasons for long backup duration, I added these options to the borg command line: -v --list --filter=AMCE --stats --show-rc -C zstd and pipe it to al log file. Therin, I found 524 files marked "A" solely from my own home directory. As far as I understand this means that 524 files are new and added to the repo within one day. That sounds plausible. But I found more than 600k files marked "M". Does that mean that these files are identified as modified and added to the repo as well? If so, this must be an error, since (almost?) all of them weren't touched for years. If all these files were processed, it would explain the (needless) long backup duration. How can I find out more? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From TSchoening at am-soft.de Wed Jun 14 02:40:41 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Wed, 14 Jun 2023 06:40:41 +0000 Subject: [Borgbackup] (too) many modified files In-Reply-To: References: Message-ID: <9a4ca8b0a16b40d58569d32552eac49d@am-soft.de> Boris Kirkorowicz wrote at Mittwoch, 14. Juni 2023 07:55: > But I found more than 600k files marked "M". Does that mean that these > files are identified as modified and added to the repo as well? > If so, this must be an error, since (almost?) all of them weren't > touched for years. If all these files were processed, it would explain > the (needless) long backup duration. The following might be of interest, as you had multiple different backup sets in the past if I remember correctly. https://borgbackup.readthedocs.io/en/stable/faq.html#it-always-chunks-all-my-files-even-unchanged-ones -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From bkborg at kirk.de Wed Jun 14 03:17:44 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Wed, 14 Jun 2023 09:17:44 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: <9a4ca8b0a16b40d58569d32552eac49d@am-soft.de> References: <9a4ca8b0a16b40d58569d32552eac49d@am-soft.de> Message-ID: Hi, Am 14.06.23 um 08:40 schrieb AM-Soft ? Thorsten Sch?ning: > Boris Kirkorowicz wrote at Mittwoch, 14. Juni 2023 07:55: > >> But I found more than 600k files marked "M". Does that mean that these >> files are identified as modified and added to the repo as well? >> If so, this must be an error, since (almost?) all of them weren't >> touched for years. If all these files were processed, it would explain >> the (needless) long backup duration. > > The following might be of interest, as you had multiple different backup sets in the > past if I remember correctly. > > https://borgbackup.readthedocs.io/en/stable/faq.html#it-always-chunks-all-my-files-even-unchanged-ones Thx. I already set BORG_FILES_CACHE_TTL=104 top shows the memory consumption of about 7,5% per borg process. There are up to 11 borg processes running simultaneously (some of them for 15 minutes, others for many hours). -- Mit freundlichem Gru? Best regards ? Kirkorowicz From jimpop at domainmail.org Wed Jun 14 12:49:42 2023 From: jimpop at domainmail.org (Jim Popovitch) Date: Wed, 14 Jun 2023 12:49:42 -0400 Subject: [Borgbackup] cache dir location Message-ID: <7f5a0dc229394517e7023102e505303cf3db19ac.camel@domainmail.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 I'm using borg v1.1.16. Is there a way to specify the cache dir location? tia, - -Jim P. -----BEGIN PGP SIGNATURE----- iQIzBAEBCAAdFiEE3RmV4WutJ2KyCS2zPcxbabkKGJ8FAmSJ76YACgkQPcxbabkK GJ9DiRAAgJytT/gxXw6jr5A0QLfY/fwcvFFCRVW9IVU4gyle6Otinb73+d+hreYV A2Nve+DeE1stsrEguWj732QN+AMnWFhgfExuW8V/J3L19mCkO3d6gz2ONc+3CSWE w0nkqzbRMd0XET77u7r4sClV9u7t3xqY4iNoU4mW6XN9dc1rLbxD16gLZEhJz6N7 32lpiPdOMr68D1nPCN/+ncV5hxUiLxs3F9n/IsbOAFhmNLrkc05NbauLTVY0vRnH 1ifsyBJVXtBTR5y3R8YQMhpaIcEhj6THWZGSm6GtcnW5Uc8BNS8moTnoXq6Jt8UW hZcrXyKlbbqZhDg0Xu/JlonLUEK5SKRdl1M/ZLEh5ZKGZ/jiGvu0F6DQD1dxGAUw y8iqrJvCaoK392Pg3g3er665pIS1Vie2Uul4sVWT4hhl0g+Q5sKQbpLsjNyqGWX1 fbnucOKq1PuR+4RnEMdVirgTBkiAoAyJ6LgnEuZ3sec13IM8eMqOteay8LmJQ13J dMFlB4eZeN9U+eqarHBblP+EIsNBCsaGKw9Uj3JsgN/JNFSPsOfG6pAPfFc/w68U iFvFaGT0W2FOTfWG00GU5NL6DGwXEvfvk8/jHvOQkoBdIFm0bC/FL8SYIWiDRiaA pTXatYZLUrYfnTndknyvNAXEDuGf+XSjhMsC3aWX+v5fGlTxclk= =MpXE -----END PGP SIGNATURE----- From TSchoening at am-soft.de Wed Jun 14 13:19:00 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Wed, 14 Jun 2023 17:19:00 +0000 Subject: [Borgbackup] cache dir location In-Reply-To: <7f5a0dc229394517e7023102e505303cf3db19ac.camel@domainmail.org> References: <7f5a0dc229394517e7023102e505303cf3db19ac.camel@domainmail.org> Message-ID: <4eefaf16f6214befad95b7f3f4702ee0@am-soft.de> Jim Popovitch via Borgbackup wrote at Mittwoch, 14. Juni 2023 18:49: I'm using borg v1.1.16. Is there a way to specify the cache dir location? There's an environment variable: > BORG_CACHE_DIR > Defaults to $BORG_BASE_DIR/.cache/borg. If BORG_BASE_DIR is not explicitly set while > XDG env var XDG_CACHE_HOME is set, then $XDG_CACHE_HOME/borg is being used instead. https://borgbackup.readthedocs.io/en/stable/usage/general.html -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From tw at waldmann-edv.de Thu Jun 15 08:22:21 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Thu, 15 Jun 2023 14:22:21 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: References: Message-ID: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> > looking for possible reasons for long backup duration, I added these > options to the borg command line: > -v --list --filter=AMCE --stats --show-rc -C zstd > and pipe it to al log file. Therin, I found 524 files marked "A" solely > from my own home directory. As far as I understand this means that 524 > files are new and added to the repo within one day. That sounds plausible. > > But I found more than 600k files marked "M". https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me "Also, the --debug-topic=files_cache option of borg create provides a lot of debug output helping to analyse why the files cache does not give its expected high performance." So, did you already use that? > Does that mean that these > files are identified as modified and added to the repo as well? Yes, M means modified. > If so, this must be an error, since (almost?) all of them weren't > touched for years. Run it with that option and it will log why it considers them modified. Could be: - content modified (and you are not aware) - metadata modified (some chown/chmod -R maybe? or xattrs changes?) - inode numbers not stable (you can switch that off in borg, if you need to) From bkborg at kirk.de Fri Jun 16 08:52:41 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Fri, 16 Jun 2023 14:52:41 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> Message-ID: Hi again, Am 15.06.23 um 14:22 schrieb Thomas Waldmann: >> looking for possible reasons for long backup duration, I added these >> options to the borg command line: >> -v --list --filter=AMCE --stats --show-rc -C zstd >> and pipe it to al log file. Therin, I found 524 files marked "A" solely >> from my own home directory. As far as I understand this means that 524 >> files are new and added to the repo within one day. That sounds plausible. >> >> But I found more than 600k files marked "M". > > https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me > > "Also, the --debug-topic=files_cache option of borg create provides a > lot of debug output helping to analyse why the files cache does not give > its expected high performance." > > So, did you already use that? > >> Does that mean that these >> files are identified as modified and added to the repo as well? > > Yes, M means modified. > >> If so, this must be an error, since (almost?) all of them weren't >> touched for years. > > Run it with that option and it will log why it considers them modified. > > Could be: > - content modified (and you are not aware) > - metadata modified (some chown/chmod -R maybe? or xattrs changes?) there are two types of reportings in the log file. The first one concerns to files that are probably changed, e.g.: > UNKNOWN: no file metadata in cache for: b'/home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4' > A /home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4 > FILES-CACHE-UPDATE: put FileCacheEntry(age=0, inode=806880962, size=22592, cmtime=1686825666505274599, chunk_ids='[1 entries]') [has ctime] <- b'/home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4' The file is shown like this at the command line: > # ls -l /home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4 > -rw------- 1 boris users 22592 15. Jun 08:22 /home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4 > # file /home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4 > /home/boris/.mozilla/firefox/97hlh6la.Boris/datareporting/archived/2023-06/1686810156551.58fe5f76-739d-4338-971b-c9adda18b5b4.main.jsonlz4: Mozilla lz4 compressed data, originally 110089 bytes The second type concerns to files that are for sure not touched for years, e.g.: > FILES-CACHE-UPDATE: put FileCacheEntry(age=0, inode=801965655, size=1623, cmtime=1686857506777454846, chunk_ids='[1 entries]') [has ctime] <- b'/home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G200.IN_' > KNOWN-CHANGED: file ctime has changed: b'/home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_' > M /home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_ Command line: > # ls -l /home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_ > -rw-rw---- 1 root users 18188 18. Aug 2001 /home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_ > # file /home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_ > /home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_: Microsoft Cabinet archive data, Windows 2000/XP setup, 18188 bytes, 1 file, at 0x2c last modified Sun, Aug 18 2001 04:55:12 +A "g711codc.ax", number 1, 2 datablocks, 0x1503 compression So maybe the Synology box does something like refreshing on it's own or something similar, if this could lead to these effects -don't know. What could cause this, and how to avoid? Or can I tell borg to ignore these files? To remember: the source Synology box is connected via iSCSI using dedicated 10GB Eth., encrypted with LUKS. The target Synology box is connected via nfs using dedicated 10 GB Eth., encrypted with ecryptfs. Borg runs with (example: my homedir): > borg create -v --debug-topic=files_cache --files-cache=mtime,size --list --filter=AMCE --stats --show-rc -C zstd $CRYPTDIR/Boris::'{now:%Y-%m-%d_%H-%M}' /home/boris > - inode numbers not stable (you can switch that off in borg, if you need to) If so, --files-cache=mtime,size should to the job? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From tw at waldmann-edv.de Sat Jun 17 12:35:30 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Sat, 17 Jun 2023 18:35:30 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> Message-ID: >> https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me >> >> "Also, the --debug-topic=files_cache option of borg create provides a >> lot of debug output helping to analyse why the files cache does not give >> its expected high performance." > there are two types of reportings in the log file. The first one > concerns to files that are probably changed, e.g.: > >> UNKNOWN: no file metadata in cache for: No, that does not mean a changed file. It is saying that it has no file metadata in the (files) cache. So either that file has never been seen before and is therefore not in the files cache (new file). Or, that file's metadata was intentionally not included into the files cache (see FAQ, files with newest timestamp in backup). Or, that file's metadata already has "aged out" of the files cache (see FAQ, files cache TTL). > The second type concerns to files that are for sure not touched for > years, e.g.: > >> KNOWN-CHANGED: file ctime has changed: >> b'/home/boris/Dokumente/Sicherungen/Joerg/SAMSUNG_SP0842N_581230084282/I386/G711CODC.AX_' The kernel / filesystem does not share your opinion: That file HAS been touched, that is the reason why it has a modified ctime timestamp. A modified ctime means that either the file metadata (like mode, owner, group, xattrs, ACLs, ...) OR the file content has changed. If the file content had changed, that could be seen via an updated mtime (but borg does not log that if it is configured to look at ctime, which is the default). > So maybe the Synology box does something like refreshing on it's own or > something similar, if this could lead to these effects -don't know. What > could cause this, and how to avoid? Or can I tell borg to ignore these > files? I have no idea about what synology boxes might do behind the scenes. But if the mtime did not change, but the ctime did, it is for sure a file metadata change (assuming that the kernel and fs works normally). >> borg create -v --debug-topic=files_cache --files-cache=mtime,size >> --list --filter=AMCE --stats --show-rc -C zstd >> $CRYPTDIR/Boris::'{now:%Y-%m-%d_%H-%M}' /home/boris Ehrm, do you have this --files-cache=mtime,size since longer? If so, ctime changes should not lead to files being detected as changed. But, after switching --files-cache from default to that non-default value, you'll need one backup to update the files cache and another one to see the files cache operating normally again. > If so, --files-cache=mtime,size should to the job? Yes. That doesn't solve the problem of your weird ctime of course, but stops using it. Using mtime is a bit less safe, as weird applications can set it to arbitrary values. From TSchoening at am-soft.de Sat Jun 17 16:20:49 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Sat, 17 Jun 2023 20:20:49 +0000 Subject: [Borgbackup] (too) many modified files In-Reply-To: References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de>, Message-ID: <774d03b2cafd4b6abb1c078d48619453@am-soft.de> Boris Kirkorowicz schrieb am Freitag, 16. Juni 2023 14:52: > To remember: the source Synology box is connected via iSCSI using > dedicated 10GB Eth., encrypted with LUKS.[...] iSCSI means Synology exposes some container file with some embedded file system only, AFAIK it doesn't easily/automatically mount that container file and accesses it to do something. The whole point of iSCSI is to make transparent storage available to some client, so the contained/exposed file system ight not even be supported by Linux or Synology. So in my opinion, I wouldn't focus on the Synology box, but the file system exposed using iSCSI and the consumer of that file system instead. Especially have a look at stable inodes and stuff, which are considered by default. SSHFS for example doesn't guarantee stable inodes, your consumed file system might not as well for some reason. > --files-cache MODE operate files cache in MODE. default: ctime,size,inode -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From howard at sci1.com Mon Jun 19 22:00:18 2023 From: howard at sci1.com (Howard Spindel) Date: Mon, 19 Jun 2023 19:00:18 -0700 Subject: [Borgbackup] italics meaning in the log? Message-ID: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> I have log entries from borg like this: M /home/howard/dd/rsyncsyno.log M/home/howard/.spamassassin/bayes_toks You'll note in the second entry /home/howard is italicized. What does this mean? Thank you. From bkborg at kirk.de Tue Jun 20 06:31:56 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Tue, 20 Jun 2023 12:31:56 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> Message-ID: <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> Hi, Am 17.06.23 um 18:35 schrieb Thomas Waldmann: >> If so, --files-cache=mtime,size should to the job? > > Yes. tried it, and met the target: duration is now less than half an hour. > That doesn't solve the problem of your weird ctime of course, but stops > using it. Using mtime is a bit less safe, as weird applications can set > it to arbitrary values. I expected that mtime is set by the file system ?? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From aleksandr.kiselyov at gmail.com Tue Jun 20 11:22:55 2023 From: aleksandr.kiselyov at gmail.com (Alexander Kiselyov) Date: Tue, 20 Jun 2023 18:22:55 +0300 Subject: [Borgbackup] (too) many modified files In-Reply-To: <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> Message-ID: On Tue, 2023-06-20 at 12:31 +0200, Boris Kirkorowicz wrote: > Hi, > > Am 17.06.23 um 18:35 schrieb Thomas Waldmann: > > > If so, --files-cache=mtime,size should to the job? > > > > Yes. > > tried it, and met the target: duration is now less than half an hour. > > > > That doesn't solve the problem of your weird ctime of course, but > > stops > > using it. Using mtime is a bit less safe, as weird applications can > > set > > it to arbitrary values. > > I expected that mtime is set by the file system ?? > > It can be easily manipulated by a userspace program, e.g. via `touch`. Regards, Alexander. From bkborg at kirk.de Tue Jun 20 11:50:28 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Tue, 20 Jun 2023 17:50:28 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> Message-ID: <4f68f984-07c5-7c04-cf00-634045b2377a@kirk.de> Hi, Am 20.06.23 um 17:22 schrieb Alexander Kiselyov: > On Tue, 2023-06-20 at 12:31 +0200, Boris Kirkorowicz wrote: >> Hi, >> >> Am 17.06.23 um 18:35 schrieb Thomas Waldmann: >>>> If so, --files-cache=mtime,size should to the job? >>> >>> Yes. >> >> tried it, and met the target: duration is now less than half an hour. >> >> >>> That doesn't solve the problem of your weird ctime of course, but >>> stops >>> using it. Using mtime is a bit less safe, as weird applications can >>> set >>> it to arbitrary values. >> >> I expected that mtime is set by the file system ?? >> >> > > It can be easily manipulated by a userspace program, e.g. via `touch`. isn't it the same with ctime? AFAICS e.g. chmod changes ctime, even if there are no changes, or chown. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From lazyvirus at gmx.com Tue Jun 20 12:00:30 2023 From: lazyvirus at gmx.com (Bzzzz) Date: Tue, 20 Jun 2023 18:00:30 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: <4f68f984-07c5-7c04-cf00-634045b2377a@kirk.de> References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> <4f68f984-07c5-7c04-cf00-634045b2377a@kirk.de> Message-ID: <20230620180030.436ff39d@msi.defcon1.lan> On Tue, 20 Jun 2023 17:50:28 +0200 Boris Kirkorowicz wrote: > >> I expected that mtime is set by the file system ?? > >> > >> > > > > It can be easily manipulated by a userspace program, e.g. via > > `touch`. Hi, > isn't it the same with ctime? AFAICS e.g. chmod changes ctime, even > if there are no changes, or chown. On a system FS mounted with noatime, touch changes all times except the birth one. Jean-yves From jdc at uwo.ca Tue Jun 20 14:46:54 2023 From: jdc at uwo.ca (Dan Christensen) Date: Tue, 20 Jun 2023 18:46:54 +0000 Subject: [Borgbackup] (too) many modified files In-Reply-To: <4f68f984-07c5-7c04-cf00-634045b2377a@kirk.de> (Boris Kirkorowicz's message of "Tue, 20 Jun 2023 17:50:28 +0200") References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> <4f68f984-07c5-7c04-cf00-634045b2377a@kirk.de> Message-ID: <87jzvy55un.fsf@uwo.ca> On Jun 20, 2023, Boris Kirkorowicz wrote: > Am 20.06.23 um 17:22 schrieb Alexander Kiselyov: >> On Tue, 2023-06-20 at 12:31 +0200, Boris Kirkorowicz wrote: >>> Am 17.06.23 um 18:35 schrieb Thomas Waldmann: >>>> Using mtime is a bit less safe, as weird applications can >>>> set it to arbitrary values. >>> >>> I expected that mtime is set by the file system ?? >> >> It can be easily manipulated by a userspace program, e.g. via `touch`. > > isn't it the same with ctime? AFAICS e.g. chmod changes ctime, even if > there are no changes, or chown. ctime is set to the *current* time when a file is changed. However, mtime can be set to an *arbitrary* time by a userspace program. So you can edit a file, and then change the mtime to what it was before the edit. That's why it isn't as safe to use mtime to determine whether a file has changed. And some "weird" programs that do this are borg, tar, touch, some version control systems, etc. For example, when you untar a tar file, tar sets the mtime to the original mtime, while the ctime is set to the current time. Dan From tw at waldmann-edv.de Wed Jun 21 06:54:49 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 21 Jun 2023 12:54:49 +0200 Subject: [Borgbackup] (too) many modified files In-Reply-To: <87jzvy55un.fsf@uwo.ca> References: <76d12cd6-5637-3699-3206-567fa92c5bf6@waldmann-edv.de> <98e1b60b-ce33-a780-1133-8a3f0475734c@kirk.de> <4f68f984-07c5-7c04-cf00-634045b2377a@kirk.de> <87jzvy55un.fsf@uwo.ca> Message-ID: <97cb40fa-611d-f5cf-9da0-6c86e7f2e6ae@waldmann-edv.de> >>>>> Using mtime is a bit less safe, as weird applications can >>>>> set it to arbitrary values. >>>> >>>> I expected that mtime is set by the file system ?? >>> >>> It can be easily manipulated by a userspace program, e.g. via `touch`. >> >> isn't it the same with ctime? AFAICS e.g. chmod changes ctime, even if >> there are no changes, or chown. > > ctime is set to the *current* time when a file is changed. However, > mtime can be set to an *arbitrary* time by a userspace program. > So you can edit a file, and then change the mtime to what it was > before the edit. That's why it isn't as safe to use mtime to determine > whether a file has changed. > > And some "weird" programs that do this are borg, tar, touch, some > version control systems, etc. > > For example, when you untar a tar file, tar sets the mtime to the > original mtime, while the ctime is set to the current time. Exactly. Would not consider restoring a file and metadata from some sort of backup or version control "weird" though. But there are really weird tools (from a backup tool's perspective): There seem to be image (like jpeg) processing tools that read an image file, modify it, write it back and then re-set the mtime to the timestamp it was before this modification. If a backup tool then would use only mtime to detect changes, it would not consider the file changed and thus not back it up, which is bad. This is why borg defaults to --files-cache=ctime,size,inode because the ctime always gets updated if any change to a file is made and the app can not "reset" the ctime. When one is forced to use mtime (because ctime is not supported by the filesystem or behaving weird), one can use --files-cache=mtime,size,inode. The size and inode check would often still trigger even for such weird tools, but there is already a small risk (like if some tool opens the original file, overwrites it with different data of same length, then resets mtime to previous value - then mtime,size,inode would all stay same and the file would be considered unchanged by borg and the content changes would not get backed up). From tw at waldmann-edv.de Wed Jun 21 06:58:56 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Wed, 21 Jun 2023 12:58:56 +0200 Subject: [Borgbackup] italics meaning in the log? In-Reply-To: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> References: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> Message-ID: <390a8d14-fec0-5749-0638-78ee879bf12e@waldmann-edv.de> > I have log entries from borg like this: > > M /home/howard/dd/rsyncsyno.log > M/home/howard/.spamassassin/bayes_toks > > You'll note in the second entry /home/howard is italicized. borg does not format output (like using italics or bold). If you can reproduce this, then please redirect the log output to a file and then analyse the file using tools like "less" or "hexdump" to see what's going on (what out if there are terminal control sequences, e.g. ANSI sequences). From howard at sci1.com Thu Jun 22 02:48:46 2023 From: howard at sci1.com (Howard Spindel) Date: Wed, 21 Jun 2023 23:48:46 -0700 Subject: [Borgbackup] italics meaning in the log? In-Reply-To: <390a8d14-fec0-5749-0638-78ee879bf12e@waldmann-edv.de> References: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> <390a8d14-fec0-5749-0638-78ee879bf12e@waldmann-edv.de> Message-ID: <8e2c2a4d-b685-6d88-0050-d47515420a07@sci1.com> Thank you for the reply, Thomas. Apparently in my first post the formatting was not preserved by this mailing list.? /home/howard below in the second line is indeed italicized. The logs I am viewing with italics are those sent to me by cron executing borg. I tried running borg from the command line instead of from cron, and added >> /logDirectory/logfile 2>&1 to the command line, but nothing is written to the log file. However, when running from the command line I did watch borg's output go by, and nothing was italicized.? Apparently cron is deciding to italicize somehow.? Can't figure this out. My theory, based on what was italicized, is that the italics indicated files that changed while borg was backing them up. Howard On 6/21/2023 3:58 AM, Thomas Waldmann wrote: >> I have log entries from borg like this: >> >> M /home/howard/dd/rsyncsyno.log >> M/home/howard/.spamassassin/bayes_toks >> >> You'll note in the second entry /home/howard is italicized. > > borg does not format output (like using italics or bold). > > If you can reproduce this, then please redirect the log output to a > file and then analyse the file using tools like "less" or "hexdump" to > see what's going on (what out if there are terminal control sequences, > e.g. ANSI sequences). > _______________________________________________ > Borgbackup mailing list > Borgbackup at python.org > https://mail.python.org/mailman/listinfo/borgbackup > From bkborg at kirk.de Thu Jun 22 03:23:22 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Thu, 22 Jun 2023 09:23:22 +0200 Subject: [Borgbackup] italics meaning in the log? In-Reply-To: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> References: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> Message-ID: <5faea227-cb18-ec25-ed21-09d5f6a50228@kirk.de> Hi, Am 20.06.23 um 04:00 schrieb Howard Spindel: > I have log entries from borg like this: > > M /home/howard/dd/rsyncsyno.log > M/home/howard/.spamassassin/bayes_toks > > You'll note in the second entry /home/howard is italicized. > > What does this mean? > > Thank you. since the output is plain text, this means most likely that your editor or viewer misinterprets the path separators '/' as italic format signs. Check if you can set an option in your editor to switch this off or try another editor/viewer. -- Mit freundlichem Gru? Best regards ? Kirkorowicz From felix.schwarz at oss.schwarz.eu Thu Jun 22 03:18:37 2023 From: felix.schwarz at oss.schwarz.eu (Felix Schwarz) Date: Thu, 22 Jun 2023 09:18:37 +0200 Subject: [Borgbackup] italics meaning in the log? In-Reply-To: <8e2c2a4d-b685-6d88-0050-d47515420a07@sci1.com> References: <235315cf-985e-e4e8-6b05-9e263fd993c5@sci1.com> <390a8d14-fec0-5749-0638-78ee879bf12e@waldmann-edv.de> <8e2c2a4d-b685-6d88-0050-d47515420a07@sci1.com> Message-ID: Am 22.06.23 um 08:48 schrieb Howard Spindel: > The logs I am viewing with italics are those sent to me by cron executing borg. User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 So you just saw the "italized" text in your email client, likely Thunderbird? If so, I think this is just the default formatting for /Thunderbird/ when it sees "/" (as present in paths). Felix From brianofish at gmail.com Sun Jun 25 08:45:49 2023 From: brianofish at gmail.com (Patrick Schaaf) Date: Sun, 25 Jun 2023 14:45:49 +0200 Subject: [Borgbackup] 2.0.0b6 apparent bug with backslashes in path Message-ID: Hi, tested today, 2.0.0b6, as an update to previously successfully used 2.0.0b4 for some root filesystem home server backups. First observation was that b6 client with b4 server wasn't working nicely at all. Probably to be expected though. But after updating them all to b6, each attempt at creating a backup, resulted in the error and backtrace below, key message being: > ValueError: unexpected backslash(es) in path 'usr/lib/systemd/system/system-systemd\\x2dcryptsetup.slice' Sorry if that's a known issue. best regards Patrick shock Local Exception shock shock Error: shock shock ValueError: unexpected backslash(es) in path 'usr/lib/systemd/system/system-systemd\\x2dcryptsetup.slice' shock shock If reporting bugs, please include the following: shock shock Traceback (most recent call last): shock File "borg/archiver/__init__.py", line 630, in main shock File "borg/archiver/__init__.py", line 536, in run shock File "borg/archiver/_common.py", line 157, in wrapper shock File "borg/archiver/create_cmd.py", line 282, in do_create shock File "borg/archiver/create_cmd.py", line 174, in create_inner shock File "borg/archiver/create_cmd.py", line 517, in _rec_walk shock File "borg/archiver/create_cmd.py", line 517, in _rec_walk shock File "borg/archiver/create_cmd.py", line 517, in _rec_walk shock [Previous line repeated 2 more times] shock File "borg/archiver/create_cmd.py", line 461, in _rec_walk shock File "borg/archiver/create_cmd.py", line 299, in _process_any shock File "borg/archive.py", line 1479, in process_file shock File "contextlib.py", line 137, in __enter__ shock File "borg/archive.py", line 1375, in create_helper shock File "src/borg/item.pyx", line 158, in borg.item.PropDict.__cinit__ shock File "src/borg/item.pyx", line 164, in borg.item.PropDict.update shock File "src/borg/item.pyx", line 232, in borg.item.PropDictProperty.__set__ shock File "borg/helpers/fs.py", line 262, in assert_sanitized_path shock File "borg/helpers/fs.py", line 233, in make_path_safe shock ValueError: unexpected backslash(es) in path 'usr/lib/systemd/system/system-systemd\\x2dcryptsetup.slice' shock shock Platform: Linux shock 6.1.28-vm #1 SMP PREEMPT_DYNAMIC Fri May 12 12:10:46 CEST 2023 x86_64 shock Linux: Unknown Linux shock Borg: 2.0.0b6 Python: CPython 3.11.2 msgpack: 1.0.5 fuse: llfuse 1.4.4 [pyfuse3,llfuse] shock PID: 18062 CWD: /root shock sys.argv: ['borg2', '-r', 'ssh://borg at shock/./rootfs', 'create', '{hostname}-{now:%Y-%m-%d-%H:%M}', '/', '--one-file-system', '--exclude-caches', '--patterns-from', '/root/.ssh/borg2_system_rootfs_patterns', '--compression=zstd,10'] shock SSH_ORIGINAL_COMMAND: None shock shock shock Command exited with non-zero status 2 shock 38.09user 4.68system 0:53.30elapsed 80%CPU (0avgtext+0avgdata 117224maxresident)k shock 0inputs+0outputs (4major+38315minor)pagefaults 0swaps -------------- next part -------------- An HTML attachment was scrubbed... URL: From tw at waldmann-edv.de Mon Jun 26 13:29:55 2023 From: tw at waldmann-edv.de (Thomas Waldmann) Date: Mon, 26 Jun 2023 19:29:55 +0200 Subject: [Borgbackup] 2.0.0b6 apparent bug with backslashes in path In-Reply-To: References: Message-ID: > First observation was that b6 client with b4 server wasn't working > nicely at all. Probably to be expected though. Yes, there is no compatibility assertion between betas (nor any beta-to-beta repo upgrade procedure). And considering that 2.0 will be a breaking release, I try to do all breaking changes for that release. So obviously stuff breaks more frequently between betas also. > > ValueError: unexpected backslash(es) in path > 'usr/lib/systemd/system/system-systemd\\x2dcryptsetup.slice' > > Sorry if that's a known issue. It's known and will be fixed in b7, see issue tracker / git history. I assumed that backslashes could only come from some processing error on windows (archived paths on windows should be "normalized" and look like windows/system32/something rather, no backslashes, no drive letters). I didn't expect anyone uses backslashes on POSIX OSes like Linux, but obviously I was wrong with that. > shock ?File "borg/helpers/fs.py", line 233, in make_path_safe > shock ValueError: unexpected backslash(es) in path > 'usr/lib/systemd/system/system-systemd\\x2dcryptsetup.slice' If you can edit the .py files, you can also just remove the backslash related check there. From TSchoening at am-soft.de Tue Jun 27 02:48:47 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Tue, 27 Jun 2023 06:48:47 +0000 Subject: [Borgbackup] 2.0.0b6 apparent bug with backslashes in path In-Reply-To: References: , Message-ID: <2d6e28440910480e98715d4e6fb4d32a@am-soft.de> Thomas Waldmann wrote at Montag, 26. Juni 2023 19:29: > I assumed that backslashes could only come from some processing error on > windows (archived paths on windows should be "normalized" and look like > windows/system32/something rather, no backslashes, no drive letters). Just out of interest: How do you handle (the same) directories on multiple different drive letters without adding the drive letter to the path? > C:\windows\system32\something > D:\windows\foobar How do paths look like in the archive for that example? I would have expected the following: > C/windows/system32/something > D/windows/foobar -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From bkborg at kirk.de Tue Jun 27 06:29:29 2023 From: bkborg at kirk.de (Boris Kirkorowicz) Date: Tue, 27 Jun 2023 12:29:29 +0200 Subject: [Borgbackup] Repo - Data Ratio Message-ID: <3848fc65-6d36-b872-a704-c974551dc287@kirk.de> Hi, after evaluating the logs, I found that the initial backup shows a ratio repo size to data size of 67,36%. That's fine. Following incremental backup however show ratio over 100% without exception, sometimes up to several thousand percent, that means that the repo size increase is more than ten times the size of the newly added data to backup. Top value is 3777%, calculated from only 9 MB new data (probably just the borg log ;-)), resulting in a 340 MB increased repo size. Ist this normal/correct/to be expected? How ist it explained? -- Mit freundlichem Gru? Best regards ? Kirkorowicz From TSchoening at am-soft.de Tue Jun 27 07:27:10 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Tue, 27 Jun 2023 11:27:10 +0000 Subject: [Borgbackup] Repo - Data Ratio In-Reply-To: <3848fc65-6d36-b872-a704-c974551dc287@kirk.de> References: <3848fc65-6d36-b872-a704-c974551dc287@kirk.de> Message-ID: Boris Kirkorowicz wrote at Dienstag, 27. Juni 2023 12:29: > Following incremental backup however show ratio over 100% without > exception, sometimes up to several thousand percent, that means that the > repo size increase is more than ten times the size of the newly added > data to backup. Top value is 3777%, calculated from only 9 MB new data > (probably just the borg log ;-)), resulting in a 340 MB increased repo size. Do a recursive directory listing of your absolute paths into some file and look at its size. Borg always stores all paths, but not necessarily with new data. When having a lot of paths, those might consume the additonal space even without new data? -- F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz From TSchoening at am-soft.de Wed Jun 28 02:26:37 2023 From: TSchoening at am-soft.de (=?utf-8?B?QU0tU29mdCDigJMgVGhvcnN0ZW4gU2Now7ZuaW5n?=) Date: Wed, 28 Jun 2023 06:26:37 +0000 Subject: [Borgbackup] WG: AW: 2.0.0b6 apparent bug with backslashes in path In-Reply-To: <879ffc28-5f50-7497-b828-61ee5f83671a@waldmann-edv.de> References: <4aa31b57e1eb47e4a5006d60eea4760f@am-soft.de>, <879ffc28-5f50-7497-b828-61ee5f83671a@waldmann-edv.de> Message-ID: <89d569f004b74cc9a2d90c2b328ec28a@am-soft.de> F?r R?ckfragen stehe ich Ihnen jederzeit zur Verf?gung. Mit freundlichen Gr??en, Thorsten Sch?ning Telefon: +49 5151 9468-55 Fax: +49 5151 9468-88 E-Mail: TSchoening at am-soft.de AM-Soft IT-Service - Bitstore Hameln GmbH Brandenburger Stra?e 7c 31789 Hameln Diese E-Mail enth?lt vertrauliche und/oder rechtlich gesch?tzte Informationen und ist ausschliesslich f?r den Adressaten bestimmt. Jeglicher Zugriff auf diese E-Mail durch andere Personen als den Adressaten ist untersagt. Wenn Sie nicht der richtige Adressat sind oder diese E-Mail irrt?mlich erhalten haben, informieren Sie bitte sofort den Absender und vernichten Sie diese E-Mail. Sollten Sie nicht der f?r diese E-Mail bestimmte Adressat sein, ist Ihnen jede Ver?ffentlichung, Vervielf?ltigung oder Weitergabe wie auch das Ergreifen oder Unterlassen von Massnahmen im Vertrauen auf erlangte Information untersagt. This e-mail may contain confidential and/or privileged information and is intended solely for the addressee. Access to this email by anyone else is unauthorized. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited and may be unlawful. Hinweise zum Datenschutz: bitstore.group/datenschutz ________________________________________ Von: Thomas Waldmann Gesendet: Dienstag, 27. Juni 2023 10:03 An: AM-Soft ? Thorsten Sch?ning Betreff: Re: AW: [Borgbackup] 2.0.0b6 apparent bug with backslashes in path > Just out of interest: How do you handle (the same) directories on multiple different > drive letters without adding the drive letter to the path? Guess windows users would create 1 archive per drive. > How do paths look like in the archive for that example? I would have expected the > following: > >> C/windows/system32/something >> D/windows/foobar Yeah, good idea, can you add it to the open PR?