Technical Notes - Q4 2025

Append-Only Backups with rclone serve restic --stdio

rsync.net users may run unix commands, remotely, over SSH like this:

or:

There is a restricted set of commands that are able to be run and because customer filesystems are mounted noexec,nosuid it is not possible to run commands that customers upload.

However, as an added defense, we also have an arguments firewall wherein we explicitly allow only specific arguments to be specified for each allowed command.

Since the inclusion of the 'rclone' command on our platform we have very intentionally disallowed the "serve" argument as we have no intention of allowing customers to run persistent processes or open sockets, answer other protocols, etc.

However ...

A number of customers, most notably Michael Alyn Miller, pointed out that the 'rclone serve restic' workflow has a --stdio modifier that causes the "serve" functions to happen over stdio without opening network sockets or spawning server processes, etc., and enables this very particular command to be run:

... which is interesting because this gives us an encrypted "append only" workflow using restic ... which is built into rclone ... which is built into our platform[1].

This is accomplished by creating a specific SSH key just for these append-only backups. This key is placed in the ~/.ssh/authorized_keys file inside your rsync.net account with a command restriction:

... which means that logins occurring with that SSH key may not run any other command than this very specific one that not only specifies --append-only but the specific repository to work on.

You will also need to create a second SSH key with no command restrictions - you'll see why as we continue below ...

On the client end (your end) you would perform an append-only backup like this:

First, initialize a restic repository in your rsync.net account using the normal SSH key that has no command restrictions:

enter password for new repository:
enter password again:
Enter passphrase for key '/home/user/.ssh/admin_key':
created restic repository 149666wedd at sftp:user@rsync.net:path/repo

Second, perform a backup of /YOUR/SOURCE/DATA to this newly initialized repository:

enter password for repository:
repository 149666wedd opened successfully, password is correct
created new cache in path/repo/.cache/restic
no parent snapshot found, will read all files

Files:        2112 new,     0 changed,     0 unmodified
Dirs:          303 new,     0 changed,     0 unmodified
Added to the repo: 869.197 MiB

You now have your first snapshot. Let's add a file, change some files, delete a file, and then do another backup:

enter password for repository:
repository 149666wedd opened successfully, password is correct
using parent snapshot dnn0629f

Files:           1 new,     1 changed,  2110 unmodified
Dirs:            0 new,     3 changed,    82 unmodified
Added to the repo: 615.911 MiB

processed 2114 files, 1.160 GiB in 0:01
snapshot a39b6628 saved

We have created a repository, uploaded data to it, and refreshed it with new data.

Now let's verify that we can see the snapshots we've created:

enter password for repository:
repository 149666wedd opened successfully, password is correct
ijnssb337c4423013b69ed833fc5514ca010160nbss223h95122fcb22h361tt7
snBSGw23hBBSj2k23d055723b2336caajsnnww23b3cc16cf88838f085bbww1kv

Finally, let's prove to ourselves that the repository is, indeed, append-only:

repository 149666wedd opened successfully, password is correct
Applying Policy: keep 1 latest snapshots
keep 1 snapshots:
ID        Time                 Host        Tags        Reasons Paths
-----------------------------------------------------------------------------------------
a39b6628  2025-03-29 20:10:44  hostname                last snapshot /YOUR/SOURCE/DATA
-----------------------------------------------------------------------------------------
1 snapshots

remove 1 snapshots:
ID        Time                 Host        Tags        Paths
--------------------------------------------------------------------------
dnn0629f  2025-03-29 20:09:54  hostname                /YOUR/SOURCE/DATA
--------------------------------------------------------------------------
1 snapshots

Remove() returned error, retrying after 720.254544ms: blob not removed, server response: 403 Forbidden (403)
Remove() returned error, retrying after 873.42004ms: blob not removed, server response: 403 Forbidden (403)
Remove() returned error, retrying after 1.054928461s: blob not removed, server response: 403 Forbidden (403)
Remove() returned error, retrying after 1.560325776s: blob not removed, server response: 403 Forbidden (403)
Remove() returned error, retrying after 3.004145903s: blob not removed, server response: 403 Forbidden (403)
 signal interrupt received, cleaning up
unable to remove  from the repository
[0:04] 0.00%  0 / 1 files deleted

What we have accomplished is a remote, encrypted backup (using restic) that can only be accessed by one of two SSH keys - a "normal" SSH key that has full control over the rsync.net account and can read/write/delete arbitrarily and a second "append-only" SSH key that cannot do anything at all except call rclone, specifically to run restic, and even more specifically in append-only mode.

This arrangement only makes sense if you keep the "normal" SSH key out of, and way from, whatever system is running these automated backups. The system that runs the backups should only have the append-only key. This way, if an attacker gains control of the source system they cannot abuse the backup configuration to destroy your remote backups at rsync.net.

[1] The restic binary is not installed on our platform and the correct way for rsync.net users to use "plain old restic" is to run it over SFTP with their rsync.net account as the SFTP endpoint.

ZFS vdev rebalancing

Let's pretend that you have a zpool with four vdevs, each of which are 90% full.

If you add a fifth vdev to this zpool, ZFS will tend to balance all new writes across all five vdevs which is a strategy that maximizes performance.

Unfortunately, if you continue to write data to this pool the first four vdevs will eventually reach 95 - 96 percent full, while the new, fifth vdev is only 5-6% full, and switch allocation strategies to "best fit" and start writing most (almost all) writes to only the new vdev.

Which is to say: your 5-vdev pool will have increased performance due to the addition of the 5th vdev for only a very short time. After that, performance will degrade markedly as you write to only the new vdev. The original four vdevs are effectively full.

One way to manage this scenario is to set a write threshold with this sysctl:

... and copy data from this zpool to itself.

For instance:

Wait for a period of light (or no) activity when you know you will be the only one writing any significant data to the pool.

Set the 'vfs.zfs.mg.noalloc_threshold' sysctl to something just above the percentage full of your existing vdevs ... in this case, if the existing vdevs are 90% full, we will attempt to grow them by 2% each, and so we will set the sysctl to '8':

... which means that at 8% free (92% full) these vdevs will stop accepting new written data.

Now find 10 TB of data on this zpool to copy (not move) back to itself.

(or, alternatively, find a 10TB dataset to 'zfs send' back to itself)

When this 10 TB of data is written to the pool it will be distributed evenly across all five vdevs.

Further, when you delete the source data (or dataset) you will be removing 2.5TB of data from each of the original vdevs which was replaced with only 2TB.

All future reads and writes of this data will now occur from five vdevs rather than four, thus increasing performance. You are, effectively, defragmenting the pool.

In this hypothetical situation our four original vdevs are now 89.5% full which means you could leave the sysctl set at '8' but this time choose a 12.5TB dataset or collection of files and copy/send that back to itself.

The condition we are trying to avoid is one where our rebalancing of data, along with other variables - or activity - on the system causes one of the existing vdevs to fill up to 95-96% at which time it will effectively stop accepting new writes.

By locking the fullness threshold with this sysctl, we can make sure we are rebalancing onto all five vdevs evenly and not overloading any one of them.

If you repeat the above procedure a few times you will achieve a manually rebalanced - and defragmented - zpool. New data will tend to write across all five vdevs and all of the data that you manually rebalanced will be newly spread across all five vdevs.

borg mount syntax

You can remotely mount a borg repo stored in your rsync.net account to a local mountpoint:

This will mount the remote borg repository, locally, in /mnt/borg. It will be mounted as a borgfs device of type "fuse".

This borg mount is read-only by default.

This HOWTO has some interesting workflows for 'borg mount' like mounting multiple repositories with a 'for' loop and 'diff'ing different mounted repositories, etc.

More Information

rsync.net publishes a wide array of support documents as well as a FAQ

rsync.net has been tested, reviewed and discussed in a variety of venues.

You, or your CEO, may find our CEO Page useful.

Please see our HIPAA, GDPR, and Sarbanes-Oxley compliance statements.

Contact info@rsync.net for more information, and answers to your questions.