Repository navigation
Update OTel docs for 1.0 telemetry config - #559
Conversation
Closes #531 Refresh the observability docs to match the 1.0 telemetry schema. This updates Helm values and runtime env semantics from the old tracing/logging/capture settings to the new exporter/traces/logs/capture layout, documents controller-owned OTEL variables, and clarifies baggage and upgrade behavior. Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Docs preview
Both are uploaded Worker versions and serve no production traffic. |
kristin-kronstain-brown
left a comment
kristin-kronstain-brown
left a comment
There was a problem hiding this comment.
Questions:
- Do the
gcp.vertex.agent.llm_requestandgcp.vertex.agent.llm_responseattributes in thetracing.mdwarning still apply on 1.0? They do not appear in the kagent repo. - kagent#2909 says the Python ADK now uses
invoke_workflowin place ofinvocation. Doestracing.md:82need the new span name?
| Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes that configuration to the agent runtimes that the controller starts. The following steps send traces to the collector that both stack guides install. To send traces to another OTLP backend, change the endpoint. | ||
|
|
||
| > [!IMPORTANT] | ||
| > The telemetry values changed shape in 1.0. `otel.tracing.*`, `otel.logging.*`, `otel.captureSensitiveContent`, and the `insecure` flag were removed, and `otel.exporter.otlp.*`, `otel.traces.*`, `otel.logs.*`, and `otel.capture.*` replace them. The chart rejects an upgrade whose values still carry a removed setting, and no Helm flag carries you across. Both `--reuse-values` and `--reset-then-reuse-values` reapply the stored values and fail in the same way. If your release still holds the old settings, write the replacements into a values file and upgrade with `--values`, as shown in the following steps. On subsequent upgrades, `--reuse-values` works again. |
There was a problem hiding this comment.
tracing.md:98: the upgrade warning. The chart has no check that rejects removed keys (the only removed-key fail in _helpers.tpl is for rbac.clusterScoped).
--reuse-valuesfails because it restores the old chart's defaults, so.tracesand.metricsare nil whencontroller-configmap.yamlrenders.--reset-then-reuse-valuesshould succeed, because it uses the new chart's defaults and ignores the old keys. That is the path kagent#2909 and Telemetry docs: rewrite for SDK-spec OTEL variables and semconv metrics (targets 1.0.0-alpha4) #531 recommend.- The thing worth warning about is that the old
otel.tracing.enabled: trueis dropped silently, so tracing turns off with no error. - The warning also contradicts step 5 (line 159), which says "Helm accepts a key that a chart does not define without an error."
| 2. Create a new Session to pick up the change, because an existing Actor keeps the configuration it started with. | ||
|
|
||
| > [!NOTE] | ||
| > Turning tracing off compiles an explicit off state rather than an absent one. The controller sets `OTEL_SDK_DISABLED` to `true` and every signal exporter to `none` in each runtime, so an agent never falls back to the OpenTelemetry SDK's own default of exporting to `localhost`. |
There was a problem hiding this comment.
tracing.md:266: the note on turning tracing off. OTEL_SDK_DISABLED=true is set only when traces, metrics, and logs are all off (TelemetryEnvironment, the !c.Enabled() branch). Both stack guides turn logs on, so for most readers of this page the SDK stays enabled. The note also sits under step 2 but describes step 1. Could the point move into step 1 instead?
| | `enabled` | Whether to export traces at all. Defaults to `false`. | | ||
| | `exporter.otlp.endpoint` | The OTLP endpoint to export to. Empty by default, which leaves the exporter on the OTel default of `localhost:4317`. | | ||
| | `traces.enabled` | Whether to export traces at all. Defaults to `false`. | | ||
| | `exporter.otlp.endpoint` | The OTLP endpoint that every signal exports to, as an `http://` or `https://` URL. An `http://` endpoint sends plaintext. Empty by default, which leaves the exporter on the OTel default of `localhost:4317`. | |
There was a problem hiding this comment.
tracing.md:134: the empty endpoint. When traces are enabled with an empty endpoint, the controller rejects the config with OTLP traces endpoint is required when traces export is enabled. It does not fall back to localhost:4317.
| The controller sets `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in every compiled runtime from `otel.captureSensitiveContent`, so setting that variable in the Harness `spec.env` field has no effect. | ||
| The controller sets `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` in every compiled runtime from `otel.capture.messageContent`, so a Harness cannot change the capture decision for itself. Naming that variable in the Harness `spec.env` field fails the `claude` and `codex` runtimes outright, and is discarded on the `kagent` runtime. For the rest of the variables that behave this way, see [Controller-owned telemetry variables](#controller-owned-telemetry-variables). | ||
|
|
||
| The controller sends the `byo` runtime no telemetry configuration, so neither setting reaches it. A `byo` image that implements OpenTelemetry itself reads whatever the Harness `spec.env` field holds. For more information, see [Tracing]({{< link path="observability/tracing#about-trace-coverage" >}}). |
There was a problem hiding this comment.
agent-harness.md:153 and audit-prompts.md:25: the byo runtime. byo/compiler.go now adds the telemetry environment, including OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, when any signal is on, and spec.env overrides it. kagent#2909 also says the controller renders these variables "into every runtime, including BYO." These lines were not in the diff, but on a page about whether prompt content leaves the runtime, "neither setting reaches it" is no longer accurate.
| Two further variables are neither owned nor passed through unchanged: | ||
|
|
||
| * `OTEL_RESOURCE_ATTRIBUTES` is merged rather than refused. The value in `spec.env` is kept, and the agent identity attributes that the controller adds win where the two name the same attribute. | ||
| * `OTEL_EXPORTER_OTLP_HEADERS` is never forwarded to a runtime, because an Actor environment holds no secrets. To add headers, export to an in-cluster collector that adds them for you. |
There was a problem hiding this comment.
agent-harness.md:176: OTEL_EXPORTER_OTLP_HEADERS. I cannot find this variable handled anywhere in go/, python/, or helm/. It is not in OwnsTelemetryEnvironment, so a value in spec.env passes through to every runtime. "An Actor environment holds no secrets" also needs a source, because CompileCredentials adds credentials to that environment. Where does this behavior come from?
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
Signed-off-by: Rachael Graham <rachael.graham@solo.io>
2 participants
Closes #531
Refresh the observability docs to match the 1.0 telemetry schema. This updates Helm values and runtime env semantics from the old tracing/logging/capture settings to the new exporter/traces/logs/capture layout, documents controller-owned OTEL variables, and clarifies baggage and upgrade behavior.