chg: [importer] file monitoring

[cvandeplas]

Implements feature request #3629 with the intent to collect your feedback and improve the code until we reach something that can be merged.

What new feature is being introduced with this PR?

• ingesting files continuously into timesketch, new files + updates in the files

Overview of changes:

Addition of command line parameters:

• monitor: to enable the monitoring feature
• monitor_state_file : optional state tracking file, as default is homedir.
• sketch_name_path_pos : the position in the path that should be used to find the sketch name
• timeline_name_path_pos: same, for the timeline name.

The sketch and timeline name path position can be used outside of the file monitoring feature.

Implemented full monitoring flow

• Added WatchdogDirectoryMonitor class (polling via PollingObserverVFS to be filesystem agnostic).
• Added _WatchdogEventHandler to enqueue filesystem events.
• Worker thread + queue to process files asynchronously.
• Initial scan (_initial_scan) to enqueue existing files at startup.
• Per-file state tracking persisted to JSON (load/save).

Potential improvements: (later)

• Multiple worker threads to improve performance. But considering we need to be careful not to overload timesketch, I'm cautious to do this immediately

Checks

• All tests succeed.
• Unit tests added.
• e2e tests added.
• Documentation updated.

Closing issues

closes #3629

[jaegeral]

Not reviewing yet, but I wonder if it make sense for you to just fork the import_client into https://github.com/google/timesketch/tree/master/contrib ?

[cvandeplas]

I considered making a separate tool, but then realized that most of the code of the timesketch_importer is to support multiple authentication methods and such generic features. So to me it felt logic to keep it together.

[jkppr]

I've added some guidance to the issue: #3629 (comment)