Log File Rotation

[ARizzo35]

Is your feature request related to a problem? Please describe.

When using a FileSink, the log file can grow unbounded until the disk is full. There is currently no way to manage this.

Describe the solution you'd like

Ideally, a setting could be provided for max file size and the data within the file could be rotated similar to how the server memory limit rotates available data sent to new subscribers. At the very least, having a max file size setting which prevented continually growing the file would help prevent filling up the disk on the system.

Describe alternatives you've considered

Another approach is to manage this outside of rerun, but I'm not sure how the file headers would get handled if the file is rotated manually.

Additional context

[Wumpf]

Adding file rotation will get tricky quickly since we'd need to start new rrd files 🤔
Sounds reasonable enough to add some opt-in monitoring/fail mechanism that can be configured on the FileSink 👍

At this point you may want to look into writing your own sink - the FileSink (the actual impl LogSink is over here for some reason)
in particular is relatively simple and since you're in Rust already, I believe it should be possible to supply your own file sinks :)

[ARizzo35]

Adding file rotation will get tricky quickly since we'd need to start new rrd files 🤔
Sounds reasonable enough to add some opt-in monitoring/fail mechanism that can be configured on the FileSink 👍

At this point you may want to look into writing your own sink - the FileSink (the actual impl LogSink is over here for some reason)
in particular is relatively simple and since you're in Rust already, I believe it should be possible to supply your own file sinks :)

Ok, I saw there's an option to output to stdout using the FileSink instead, but I'm not sure how the file headers would get managed in that case. Are there any file headers? Do they always get output first in some predetermined format by the FileSink when it is initialized?

[Wumpf]

Well as mentioned that's the hard part - a readable rrd starts with a small header and usually has some store info message early on. I think though we can actually just nit arbitrary log messages together, but I don't think there's any meaningful way to open them if they don't each start with a store info message first.
The header itself is handled here https://github.com/rerun-io/rerun/blob/0.29.1/crates/store/re_log_encoding/src/rrd/file_sink.rs#L93