stats: process RPC stats/tracing in health and ORCA producers only if a handler is configured

[mbissa]

This resolves an issue where below log statement keeps printing repeatedly:
"ERROR: [otel-plugin] ctx passed into client side stats handler metrics event handling has no client attempt data present".
When new stream is created for health/orca producers, stats and tracing is not setup. However, this fact is ignored during RPC and an error logs is printed to denote that stats cannot be handled. We will enable stream to have its own reference to the stats handler and only process per RPC implementation when it is present (like in case of regular data streams).

RELEASE NOTES:

• stats: only process RPC stats/tracing in health and ORCA producers if a handler is configured, preventing unnecessary error logging

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.28%. Comparing base (49e224f) to head (8c029f6).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8874      +/-   ##
==========================================
+ Coverage   83.15%   83.28%   +0.12%     
==========================================
  Files         414      414              
  Lines       32751    32752       +1     
==========================================
+ Hits        27235    27278      +43     
+ Misses       4096     4070      -26     
+ Partials     1420     1404      -16     

Files with missing lines	Coverage Δ
internal/transport/client_stream.go	100.00% <ø> (ø)
internal/transport/http2_client.go	92.35% <100.00%> (+0.20%)	⬆️
internal/transport/transport.go	89.06% <ø> (ø)
stream.go	81.87% <100.00%> (ø)

... and 16 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[easwars]

Curious, how did you run into this? Is there an issue for this?

[easwars]

When new stream is created for health/orca producers, stats and tracing is not setup

Can you please elaborate on this?

My understanding is that when the opentelemtry.DialOption is specified at channel creation time, the gRPC channel is configured with the oTel stats handler. And this is stored in the cc.statsHandler field which is what is what gets propagated to the RPC attempt struct csAttempt in newAttemptLocked(). So, this information should also be available when creating the health or ORCA stream, right? Or is it the case that statstracing should be disabled on these streams?

I think it would be good to open an issue that documents all the details and the approach taken (and why this is the preferred approach, if there are other approached available at all). Thanks.

[easwars]

Is this how the health and ORCA streams get a nil stats handler? We want to eliminate this duplication between the regular client stream and the non-retry client stream at some point. We tried to prioritize that work for Q1, but it didn't happen. This has been a maintenance burden for a while now, and a source of hard to find bugs.