[Borgbackup] Stat of looots of files

[Thomas Waldmann]

> hds: 16 ST10000NM0256 Seagate 10TB 7.2K SAS 12GBps 3.5 (excluding the
> 2 for ZFS cache)

The 2 for ZFS cache are SSDs?

> I also explored the "zfs diff" to compare snapshots and try to submit
> to Borg just the difference to process.
> But it's the same story. ZFS diff walks the "two trees" in single
> thread way and compare them. So, no gain.

Not sure if one could use "zfs send" somehow?

> I'm also thinking to write a stupid demon with inotify to keep a list
> to offer to Borg.

borg always creates full backups - all files are contained in the backup 
archive (even if a file is unchanged, it will be in the archive [== it 
will have a metadata item in the archive metadata stream] - borg just 
will not store the unchanged content chunks of it again into the repo).

Therefore, knowing a list of changed files does not help as borg will 
look at all files anyway to create that **full** metadata stream.

If you'ld go away from that, it would totally change semantics from 
"always full" to "full or incremental/differential" and you'ld have to 
be very careful when pruning to not kill stuff you still need.

But another thing that will come very soon (1.1.16 release) might help 
you a bit with speed: if you neither need to back up bsdflags, nor 
xattrs, nor ACLs, you can run borg create with --noxattrs --noacls 
--nobsdflags options and that will speed it up. How much very much 
depends, can be a little or much (see that winbind nss ticket).

Cheers, Thomas

-- 

GPG ID: 9F88FB52FAF7B393
GPG FP: 6D5B EF9A DD20 7580 5747 B70F 9F88 FB52 FAF7 B393