Documentation is insufficient to implement OpenTelemetry for Agent Engine properly. #1215
Description
Describe the bug
When deploying an agent to Agent Engine, there is a very specific order of operations and precise keys/values required for OpenTelemetry traces to be emitted properly, ingested, and to appear on the Vertex AI dashboard. For example, tracing init MUST NOT be triggered outside of the app set_up() call.
If values don't align or init happens out of order, or the infrastructure is not configured (terraformed) precisely, you end up with blank dashboards for your agent, and no traces.
This information is encoded (sideways) in the agent starter pack, but not explained... and touching the code to customize it leads to easy and mysterious breakage because the contract is not clear.
To Reproduce
Try to setup telemetry without the starter pack, and only these docs. It is nearly impossible.
Expected behavior
One should be able to implement OTEL for Agent Engine without a starter pack template, simply by following the documentation.
Here are the things which would help:
- Documenting clearly the init sequence would be VERY helpful.
- Documenting the canonical data flows to support the dashboards via Log Analyzer
- An architecture diagram of the canonical infrastructure configuration
Bonus would be: "look at the OTEL agent in https://github.com/google/adk-samples" which would include things like header tracing over MCP as a bonus...
Versions
- ADK version: 1.23