Borg 2 - BorgStore: Question About Performance

[zR-JB]

BorgStore is the name of the library used by Borg 2 (master branch) that implements a key/value store and supports different kinds of backends: fs, sftp, rclone, and maybe also S3 directly soon.

In another discussion, @ThomasWaldmann mentioned the different backends for BorgStore. I was wondering how the performance compares between them.
Let’s say we have two servers connected via a 10 Gbit network, both running btrfs and have a modern CPU with very good single core performance (think of a game server). One server will start an encrypted and compressed backup (10 TB) to the other server. What kind of performance can we expect when using SSH, SFTP, or rclone as the backend?

• Will SSH be the fastest, since we can run borg serve on the receiving side and offload some of the work (not sure if borg2 does that)?
• Is there a difference between just using SSH, and using SSH with the Borg binary on the receiving server (i.e., running borg serve)?
• Is rclone a better option for transferring large amounts of data, or is it a less ideal option just for platforms that do not support direct ssh access?

I’d appreciate any insights or experiences regarding performance with these different backends and what the limiting factor is for each of them. Thanks!

[ThomasWaldmann]

As of 2.0.0b18: speed local > ssh > sftp/rclone/s3 (my best guess, I didn't measure it).

Of course it does not only depend on the protocol, but also on the data backed up, the connection speed AND latency.

I have recently started on implementing packs, they likely will change performance characteristics significantly, especially on high latency connections and with small chunks.

[cicone]

It would be interesting to know more about the packs feature design and expected performance gains. I have been using borg 1.x for backing up locally and in cloud entire filesystems for years, with two stages:

1. borg create from RO snapshot to local repo (portable SSD)
2. rclone sync for the whole repo to a remote provider (pCloud)

This worked very well, with good performances, with little tweaks to rclone (and none to borg) it was possible to efficiently transfer on slow and fast networks with little overhead.

The only drawback which comes to mind was that compacting segments would require uploading a lot more data than it was actually new/updated.

I was excited to read about the new features provided by borgstore in borg2, including native rclone support, and gave it a try. The results were disappointing:

1. the workflow I was used to did not work, after creating the local repo without issue, rclone was unable to sync the repository to remote, due to backend limitations. the huge number of files (>1M, compared to 3k for borg1) caused even simple listing commands (API requests) issued by rclone to fail, with backend error, or take an extremely long time. To bring a little perspective, what would have taken about 1hr to create and 6-7hr to transfer took less time to create (locally) but after several days was not even close to 50% of transfer.

2. using directly the remote rclone backend apparently was working, but again very slow and impractical for this use case.

With default params, millions of chunks get created, corresponding to millions of files. This does affect local filesystem (du taking 6s instead of ms) but the performances are enough to ignore the penalty for the added benefits. The same may apply in cooperative remotes with borg installed. But for cloud storage with http APIs it is unlikely to work well as is. Maybe packs will address that?

[ThomasWaldmann]

Packs will likely solve these issues as they will reduce the borgstore object count significantly (fewer, larger files, less roundtrips, less storage overhead).

The packs will be assembled client-side (that's different from borg 1.x) and then put into borgstore. Still need to work out lots of the details, this will be a quite big change and take a while...

[cicone]

Thank you, the concept seems promising.