[Borgbackup] first backup of large amounts

[Boris Kirkorowicz]

Hello,

Am 27.01.23 um 04:25 schrieb William Kenworthy:
> 1. Create the repo. Backup a subset of the data, next time increase the
> size of the subset - the already backuped up data will use the fast
> heuristics, though any new data will be added at the normal first time
> rate.  Repeat until the whole dataset is covered.
> 
> 2. create a largish number of smaller repos in parallel (ideally using a
> number of hosts to maximise throughput - I am doing ~20 at once for
> different hosts/data sets into separate repos on a single moosefs file
> system, MUCH faster than trying to do it serially) - extract and add
> back into a single main repo. This last bit will be slow.
> 
> 3. create a largish number of smaller repos as above and keep them that
> way and manage by scripts - recommended.  Will be more reliable, faster
> and less susceptible to corruption across the whole data set - while
> corruption is rare, it DOES happen - and the time to recover a large,
> single repo is really really really large to the point its usually
> quicker to recreate it!

thanks, good point. I'll follow this and see how to split the data into 
reasonable chunks. But it looks like there will be some parts that 
should not be split and will still remain very large.


> 
> BillK
> 
> 
> On 26/1/23 22:34, Boris Kirkorowicz wrote:
>> Hello,
>> I want to backup a somewhat larger amount of data, approximately 35
>> TB. My first tries let me assume that 70 GB need about 1 hour to
>> backup. These 35 TB would take about 2~3 weeks in the given
>> environment. That means that the server would not be available during
>> an unacceptable long time. So I wonder if it would be possible to
>> split the backup into pieces of max. 400 GB (~ 6 hours) while getting
>> a single repository at the end.
>>
>> What would be the best way to begin the backup?
>>
>>
>>
> _______________________________________________
> Borgbackup mailing list
> Borgbackup at python.org
> https://mail.python.org/mailman/listinfo/borgbackup

-- 
Mit freundlichem Gruß                                 Best regards
                                  Kirkorowicz