[borgbackup] Borg speed tuning on large files

[Alex Gorbachev]

On Tue, Aug 25, 2015 at 4:30 AM, Thomas Waldmann <tw at waldmann-edv.de> wrote:

> On 08/25/2015 03:35 AM, Alex Gorbachev wrote:
> > Hello, new challenge on performance - running a machine with two 4
> > core 2 GHz CPUs, 32 GB RAM and pretty fast disks.  Trying to run a
> > dedup backup of 14 files, 20-150 GB in size with a total of about 2TB.
> >
> > When borg runs I see IO rates via iostat that are far below the
> > storage subsystem capabilities.  top shows 97-99% load for borg and
> > around 100 MB RSS.
>
> Do you use compression or encryption?
>

No encryption.  Tested with compression and without - observed the behavior
you described below, but the volume of data pretty much requires
compression to properly function.  I tried compression levels of 0 and 3.


>
> I am wondering a bit that you get so close to 100% as due to the
> single-threaded and relatively simple way of internal working of the
> current release, one usually only gets that close when using high
> compression (and in that case, the high compression can be a bottleneck)
> or no cpu-accelerated encryption (or both).
>
> If you use encryption and openssl can't use AES-NI (because the cpu does
> not support it or the drivers are not loaded), that can also slow down
> things.
>
> So, for first speed tests, I'ld recommend no or fast compression and no
> encryption. With 0.24 release that is --compression 0 (default) or
> --compression 1.
>
> Next release will add super fast lz4 compression (which I think you will
> like if your I/O system is rather fast), I hope I can release it in a
> few days.
>

Oh can't wait - this is what ZFS uses and ours is plenty fast.  I also
realized that we are going from a compressed ZFS to its snapshot and then
to uncompressed target destination with borg (as borg already compresses),
so there is overhead on ZFS decompression...but that should run on another
core.


>
> Keep an eye on borg's cpu load and get it to a bit lower value (maybe
> 50-80%),
> so it is in the sweet spot in the middle of being I/O bound and being
> CPU bound.
>

I turned of hyperthreading and enabled aggressive CPU power mode, running a
test, which will take about 25+ hours on the 2TB


>
> Also, as I've already said, I am sorry that 0.24 had broken
> --chunker-params parameter parsing, so do not use that right now, this
> will also be fixed in next release asap.
>

Thanks, I compiled right away from the git tree and no problems there,
thank you for the fast response


>
> > I am assuming the bottleneck is the CPU as borg is single threaded.
> > Is there anything we could do to speed the process up though - more
> > RAM caching somehow?
>
> I am working on a multithreaded implementation (which is not trivial and
> not expected to be finished soon) which can use the CPU cores and I/O
> capabilities of a system much better.
>

Completely understood, and I am assuming starting multiple borg processes
in parallel for each file is not a good idea?


>
> If you want to play with it, it is in multithreading branch of the repo,
> but do NOT use that for real backups, it has still failing tests and
> also the crypto might be unsecure there.
>
> I've seen > 300% CPU load with that code on a dual-core cpu with
> hyperthreading and also the wallclock runtime was better than with
> single-threaded code (but not 3x better, there is also some overhead).
>
> --
>
> GPG Fingerprint: 6D5B EF9A DD20 7580 5747  B70F 9F88 FB52 FAF7 B393
> Encrypted E-Mail is preferred / Verschluesselte E-Mail wird bevorzugt.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/borgbackup/attachments/20150825/dde7aab7/attachment.html>