Append-only ssh-based backups

[Stebalien]

I'm using a Hetzner (I know, probably not the best) storage box and have setup restic to backup with to rclone over ssh by following this tutorial: https://ruderich.org/simon/notes/append-only-backups-with-restic-and-rclone

Basically:

1. Restic starts a remote append-only rclone instance with ssh, instructing that instance to communicate via stdin/stdout instead of listening on a port.
2. Restic then communicates that rclone instance via stdin/stdout over ssh.

This lets me configure a "restricted" shell on the remote machine that only runs a specific rclone serve restic command in append-only mode.

Is there any way to replicate this behavior with rustic? If not, is there a better way to do this (backblaze, I guess...)?

[aawsome]

In fact, using storage backends via rclone is always just these two steps:

• starting rclone serve restic
• using the rest: backend to access the rest server provided by rclone

Now, rustic doesn't support the possibility to access rclone via stdio. Using TCP is generally the better and easier way, but has the drawback that you have to handle ports and maybe network security.

In your case this means, you should start rclone via ssh with port-forwarding the provided rest-port to a local port. Then just use rustic with rest-backend.
I think we can also add something like a rclone-command/rest-port to rustic to support this within the rustic configuration. Currently you must run the above by hand.

[Stebalien]

Hm. Unfortunately I can't do port forwarding in this situation (can't forward ports on the remote host. I'd have to proxy TCP. Is there any reason TCP is easier/better? That tends to have quite a bit of overhead, requires figuring out ports, and has security implications (websites in the local browser can likely access a running rclone instance this way unless you're protecting it somehow).

[aawsome]

Don't know if this works with the Hetzner StoragBox, but I thought about using ssh to forward the port. Something like
ssh -L 8000:localhost:8001 user@storagebox rclone serve restic --addr localhost:8001 my_rclone_path
and then use rest:http://localhost:8000 as backend for rustic.

And - yes, you are right, that would everyone on your local host allow access to your storage using port 8000. But given that you would use rclone with append-only and everything in the repository is encrypted that wouldn't be too risky, IMO.

Note that I just created rustic-rs/rustic_core#139.
Once that PR is merged and a new rustic_core/rustic is released, you could do the above using

[repository]
repository = "rclone:my-rclone-path
[repository.options]
rclone-command = "ssh -L 8000:localhost:8001 user@storagebox rclone serve restic --addr localhost:8001"
rest-url = "http://localhost:8000"
use-password = "false"

which would also start the ssh connection and close it after operation.

(Note that you could also try to let rustic set and use a randomly generated password by setting use-password to true, but this would involve forwarding env variables to your remote and I don't actually know if this would work with you StorageBox)

[aawsome]

Ah, and of course that should also work with a restricted shell - given that you can get the port-forwarding working.

In case there is no hope for the ssh port-forwarding, you can only open an issue to add the possibility to use rclone over stdio to rustic. When I implemented the rclone backend I also took a look how to do this and finally decided that I'd rather implement something to secure the TCP connection by randomly generated user/password - I didn't find an ease way or rust crate to do the stdio trick. Moreover it seemed a bit restricted to me in case we want to talk about parallel connections to the storage backend...
But maybe there is someone out who could propose a PR?!?

[Stebalien]

Thank you! The use-password feature resolves my security concerns on localhost (although I'd be concerned about running that on, e.g., a shared hosting box given that the password would be on the commandline).

Unfortunately, hetzner is rather restrictive (for good reason) and only allows rclone to be started as rclone serve restic --stdin (on storage boxes, that is).

I didn't find an ease way or rust crate to do the stdio trick. Moreover it seemed a bit restricted to me in case we want to talk about parallel connections to the storage backend...
But maybe there is someone out who could propose a PR?!?

I may take a look if I find some time. You're right about the restricted parallelism, although I assume we can rely on HTTP pipelining to avoid the round-trip penalty, at least.

It might actually be simpler to just write a reverse http proxy (http requests in, pipelined HTTP out over stdin/out.)