Prometheus Metrics for TesseraCT

[pimvanpelt]

The underlying Tessera and this TesseraCT project is using OpenTelemetry for logging and tracing, which is tied to GCP as a runtime environment. For self-hosted environments, and even for GCP/AWS hosted environments, having telemetry for observability is vital. To run TesseraCT (I am interested in operating a POSIX ctlog), having a /metrics endpoint to scrape with Prometheus is a requirement.

Our colleagues who develop Sunlight have done this very well. A sampling of their telemetry which I think would be a useful starting point for TesseraCT:

addchain_requests_total
addchain_wait_seconds
addchain_wait_seconds_count
addchain_wait_seconds_sum
cache_get_duration_seconds
cache_get_duration_seconds_count
cache_get_duration_seconds_sum
cache_put_duration_seconds
cache_put_duration_seconds_count
cache_put_duration_seconds_sum
cache_put_errors_total
config_notafter_end_timestamp_seconds
config_notafter_start_timestamp_seconds
config_roots_total
fs_op_duration_seconds
fs_op_duration_seconds_count
fs_op_duration_seconds_sum
http_in_flight_requests
http_request_duration_seconds
http_request_duration_seconds_count
http_request_duration_seconds_sum
http_requests_total
issuers_cache_total
sequencing_data_tiles_bytes
sequencing_data_tiles_bytes_count
sequencing_data_tiles_bytes_sum
sequencing_data_tiles_gzip_bytes
sequencing_data_tiles_gzip_bytes_count
sequencing_data_tiles_gzip_bytes_sum
sequencing_duration_seconds
sequencing_duration_seconds_count
sequencing_duration_seconds_sum
sequencing_leaf_bytes
sequencing_leaf_bytes_count
sequencing_leaf_bytes_sum
sequencing_pool_entries
sequencing_pool_entries_count
sequencing_pool_entries_sum
sequencing_rounds_total
sequencing_uploaded_tiles_total
tree_size_leaves_total
tree_timestamp_seconds

A few other things that could be useful:

• Badger DB telemetry.
• Knowing about duplicates will help.
• A notion of checkpoint target (1500ms) and actual (could be larger, signalling contention).

[AlCutter]

Hi @pimvanpelt,

Thanks for the request!
We have OTEL instrumentation integrated throughout tesseract and tessera which covers a large chunk of these metrics, but the POSIX binary doesn't have a means to export that at the moment.

I've just sent #482 which brings that into line with the other binaries.

Prometheus does have native support for OTEL/OTLP, with instructions on how to configure that here: https://prometheus.io/docs/guides/opentelemetry/ (TL/DR: you set a flag on prometheus to enable that, and then export some env vars when running tesseract to tell it where to push metrics to).

I've just tested this locally and it seems to work; here's a rough-and-ready grafana dashboard I knocked up using Prometheus as a datasource, with metrics being pushed using the instructions above:

LMK if this works for you, or you need something else.

We can also easily add more metrics.