Periodic backups for researchers

[oscgonfer]

Together with @timcowlishaw we have come up with the idea to make backups of certain devices on an S3 bucket (or other similar storage option) for research devices which their deletion would be critical (research projects, for instance).

Ideally, a device could have a simple flag available for researchers that enables a mechanism that periodically stores data on a bucket: /device/sensor/date.{csv,parquet} or /user/device/date.{csv,parquet}

That bucket would for now be on our premises. Agreeing to the backup would mean complications in terms of GDPR, and the user would have to grant permission for processing their data explicitly.

Later on:
1.Signed, timed URLS could be generated for download
2. Endpoint for requesting URLs could be added
3. Optional "custom" bucket could be added to send data to a custom one

@timcowlishaw am I missing anything from our conversation?

[oscgonfer]

Additionally, we should consider not only this as a problem of data deletion, but also access for devices with such long timeseries where requesting through the API is an insane amount of data.

[pral2a]

I’m adding this here because it hasn’t been clarified in the issue, but I believe it’s well understood by everyone that the mechanism we are proposing is very close to the existing CSV download option.

In that flow, we process the time series through a sidekick job that generates a CSV file, store it in an S3 bucket, and then email the URL to the user. What we’re suggesting here is essentially the same process: extend the initial case to make it more generic, and add the specific features needed to support this case.

@oscgonfer If I’m following correctly, the process would be scheduled instead of user triggered, with the URLs either emailed to users or exposed through the web interface.

[oscgonfer]

@pral2a exactly. Thanks for the clarifications.

I think scheduling could be done with some load balancing (at times with fewer backups, and/or little API requests). The user would see a url after making the first trigger, which could be sent via email when it's ready. Periodic updates may not need the email to update the user.

Making this periodically leaves us with the question of how we store it. For instance, a backup per day and a simple csv would do. However, I'd advocate for more sofisticated option that would make use of a file format that can be incrementally grown, such as parquet, without having to rewrite the whole thing and making it possible for more advanced analytics directly with existing tools.

[pral2a]

Given that what we are proposing here might turn out to be a bit more complex, and without the intention of adding unnecessary overhead, I would also suggest evaluating the possibility of building our backup system on top of InfluxDB.

When we originally built the platform, advanced time series databases like InfluxDB were not available. If we ensure that data written to KairosDB is also duplicated into InfluxDB, we could potentially leverage its built-in features to provide users with download and backup options.

This is worth exploring, at least on paper, as part of future roadmap discussions. @oscgonfer added for future reference:

• InfluxDB 2.x provides a function to export subsets of data in Line Protocol format.
• InfluxDB 3.x natively stores data as Parquet files. It would be useful to assess whether this functionality is available in the open source “Core” edition or only in the enterprise offering.
• It may also be valuable to review alternatives such as Apache IoTDB, in particular its data synchronisation feature.