Many dimensions with `http.client.open_connections`

[JamesNK]

Description

See https://www.reddit.com/r/dotnet/comments/1pwkqfl/looking_for_example_code_of_opentelemetry_tracing/nw4k1wy/ and Azure/azure-sdk-for-net#52674

The tags on the metric should be low cardinality. High cardinality tags creates many dimensions.

Many dimensions are bad:

• Some tooling will stop recording values when dimension limit is reached (prevent excessive memory use)
• Observability platforms can charge per-dimension stored, so many dimensions could create a large bill

Reproduction Steps

Send requests to many servers. Each server adds multiple dimensions to the metric because "server.address", "server.port" and "network.peer.address" tags will be different for each server.

Expected behavior

Constrained number of dimensions.

Actual behavior

Unconstrained number of dimensions.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

[kevbry]

Thanks for creating this!

IMO the issue with http.client.open_connections isn't necessarily that there are too many dimensions, it's that it continues to emit measurements even after those measurements become meaningless. For http.client.open_connections, measurements are emitted when both the idle and active counts for a given endpoint are zero. In other words, making a single http request to a given endpoint starts two new endless streams of measurements (one for each of the possible values of http.connection.state), all but the first dozen or so of which always have a count of zero.

Reducing the cardinality by clipping out the remote IP is the only way to reduce the noise right now because of the lack of filtering APIs in otel, but a better solution would be to stop emitting measurements once the connection left the pool and stopped being an open_connection.

[Clockwork-Muse]

because of the lack of filtering APIs in otel

It's true that there isn't a "standard" API, for this, but this is one of the things you can do if you deploy a collector; eg configure the filter processor.

For cloud tracing backends, it may also be possible to separately configure them to drop or otherwise exclude the records in such a way that they aren't counted as a chargeable event.

[kevbry]

Sorry, I should have been more specific - I meant in the .net otel library, which doesn't have APIs allowing filtering of metrics.

With a quick read through the docs on filter processor I don't see how to express what I need to do here, which is omit measurements which match on all tags aside from http.connection.state and where both counts are zero. Even if it is possible, I'd imagine it's going to be computationally expensive to do so.

[ManickaP]

So the tags are defined by OTel: https://opentelemetry.io/docs/specs/semconv/dotnet/dotnet-http-metrics/#metric-httpclientopen_connections
Address is only "recommended", otoh port is mandatory if it's not the default value and having port specified without address seems wrong to me.

@noahfalk is there a way to actually stop emitting the counter once it's been started?
If so, I don't see an issue with us doing that. If not, maybe we should explore a way how to do that.

Disclaimer, I was not present during the OTel discussions, so feel free to correct me on anything.

[noahfalk]

@noahfalk is there a way to actually stop emitting the counter once it's been started?
If so, I don't see an issue with us doing that. If not, maybe we should explore a way how to do that.

One option is to switch from UpDownCounter to ObservableUpDownCounter. This will give you a callback every time the metric is going to be emitted. The http client code needs to provide the value of the metric for each tag combination you want to report. Any combinations you don't emit during the callback (for example all the logically zero-valued ones for old connections) shouldn't be transmitted or stored.

The other option, more of a workaround IMO, would be encouraging users to use cardinality limits to ensure that not too many different tag combinations are sending data at the same time. I don't recall if the OTel SDK is sophisticated enough to prioritize the more recently added tag combinations which users would care about but at minimum it can ensure the total amount of data is constrained.

HTH!

[JamesNK]

So the tags are defined by OTel: opentelemetry.io/docs/specs/semconv/dotnet/dotnet-http-metrics#metric-httpclientopen_connections Address is only "recommended", otoh port is mandatory if it's not the default value and having port specified without address seems wrong to me.

The semantic conventions aren't perfect. They might not be aware this is causing issues. I suggest you create an issue in their GH repository to discuss the problem.

[ManickaP]

So the OTel states which type of instrument to use: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/dotnet/dotnet-http-metrics.md#metric-httpclientopen_connections
I do not understand the diffrences enough to be able to say whether we can change it to its "observable" version. Can this have an impact on the listeners? Can this influence the observation tools and visualization?

I still think it would be valuable to be able to "remove" the irrelevant values/dimensions from the reported results / observed counters. @noahfalk you already have an issue on the similar note #83822 it seems.

And I can file this up with OTel, but IMHO the address is valuable dimmension to this counter, so I'd rather see this resolved in less radical way.

Oh and @JamesNK, could the original reporter try out the cardility limits suggestion from Noah?

[noahfalk]

I do not understand the diffrences enough to be able to say whether we can change it to its "observable" version.

The difference between Counter/UpDownCounter/Histogram are important and determines the shape of the emitted data and how down-stream consumers would interpret it. The difference between the Observable and non-Observable forms however are (with some small caveats*) implementation choices that you are free to switch between. I don't expect anyone will complain if you switched this to Observable and I've never interpreted the semantic convention that says UpDownCounter to be mandating the specifically the non-observable form of it. Its an awkward naming issue where the name for the set of {UpDownCounter, ObservableUpDownCounter} is also UpDownCounter so its hard to tell when someone is refering equally to both or specifically to the non-observable form.

(*) The caveats I'm aware of if you are curious:

• The metrics exemplar feature only provides exemplars for the non-observable form of the instruments. Its possible someone out there will be sad about losing their exemplars on this metric but I don't think its a big enough concern to avoid the ObservableUpDownCounter.
• By using the ObservableUpDownCounter's capability to directly control the aggregation you can trim tag combinations which is of course visible to the downstream consumers as well, but I struggle to imagine why anyone would be sad to lose their infinite streams of zeros.