feat(observability): [WIP] implement universal 4-path OpenTelemetry tracing

[chalmerlowe]

feat(observability): implement universal 4-path OpenTelemetry tracing

Problems Solved

Google Cloud Python client libraries support four communication paths: synchronous gRPC, asynchronous gRPC, synchronous REST (HTTP), and asynchronous REST (HTTP). Previously, distributed OpenTelemetry tracing was only wired for synchronous gRPC calls, leaving asynchronous and HTTP communications untraced. Additionally, earlier drafts of asynchronous tracing attempted to modify gRPC channels after creation, which violated the immutability rules of the underlying Python gRPC library, and did not consistently forward client options across all transport classes.

Solutions

This pull request provides a unified, cross-transport tracing implementation:

1. Universal 4-Transport Support:

   • Synchronous gRPC (GrpcTransport): Continues using OpenTelemetry gRPC channel interceptors.
   • Asynchronous gRPC (GrpcAsyncIOTransport): Supplies OpenTelemetry interceptors directly during channel creation, respecting the immutable design of asynchronous gRPC channels.
   • Synchronous REST (RestTransport): Adds wire span tracking around HTTP requests with automatic W3C trace context header injection (traceparent).
   • Asynchronous REST (AsyncRestTransport): Integrates HTTP wire span tracking and async context lifecycle handling with W3C header propagation.

2. Refined Transport Contracts & Cleanup:

   • Replaced per-instance signature inspection with module-level constants to eliminate overhead during method invocation.
   • Configured BaseTransport.kind to safely return an empty string by default instead of raising an exception.
   • Updated REST and base transport initializers to accept and store client options, ensuring options are forwarded uniformly across all transport classes.
   • Resolved an issue in asynchronous method wrappers where caller-specified compression settings were not correctly evaluated.
   • Added task cancellation handling (asyncio.CancelledError) so spans are closed accurately when asynchronous tasks are cancelled.

Notes for Reviewers

• This pull request targets template definitions in packages/gapic-generator and core helper functions in packages/google-api-core.
• Per team coordination, generated client library code and integration goldens are not included in this change and will be updated in a follow-up step.
• All additions maintain backward compatibility and preserve zero overhead when tracing is disabled.

[gemini-code-assist]

Code Review

This pull request implements OpenTelemetry tracing capabilities across gRPC and REST transports for generated clients. It introduces _observability modules, updates transport constructors to accept client_options, and wires tracing interceptors into the transport layers. The reviewer identified a critical issue in the _shared_macros.j2 template where start_http_span was called with incorrect arguments, which would cause a runtime TypeError. The suggested fix involves passing a properly constructed request object to the tracing utility.

[gemini-code-assist]

The current call to _observability.start_http_span passes client_options as the first positional argument (which maps to request in the function signature) and passes method, url, headers, and body as keyword arguments. However, start_http_span does not accept these keyword arguments and expects a request object with those attributes. This will result in a TypeError at runtime when tracing is enabled.

To fix this, dynamically construct a lightweight request object using Python's built-in type() constructor and pass it as the request argument, while correctly passing url_template and client_options as keyword arguments.

        with _observability.start_http_span(
            type("Request", (), {"method": method, "url": url, "headers": headers, "body": body})(),
            url_template=uri,
            client_options=client_options,
        ) as span:

[chalmerlowe]

Note

As context for the reviewer:

For native methods (like Echo or GetSecret), the generator reads the service's .proto file directly, so constructing the name in the template is straightforward:

method_name="{{ '.'.join(method.meta.address.package) }}.{{ service.name }}/{{ method.name }}"

However, mixins don't live in the service’s proto. Mixin methods (GetOperation, GetIamPolicy, ListLocations) are synthetic—they are injected by the generator from the static catalog in gapic/schema/mixins.py

Without a name attribute, any mixin call (like polling an operation or checking IAM permissions) would be unable to start an OpenTelemetry method span, or would emit an unknown/nameless span that failed our contract checks.

[chalmerlowe]

The focus for this file is to get fundamental/core elements into the Base class to:

• Help eliminate some checks that we were originally considering placing in the Transport classes, etc.
• Cut down on some of the boilerplate in the Transport classes

[chalmerlowe]

Comment for Reviewers:

This logic got moved to the base transport.

[daniel-sanche]

There's a lot I haven't looked at yet, but wanted to get some of my first comments out

[daniel-sanche]

nit: This works, but it seems like the name component is duplicated in both

Should we store a package_name instead, then build this as {package_name}/{name}?

[daniel-sanche]

   # Excluded from coverage because our CI and testing environments always install
   # a modern version of google-api-core that supports tracing.

Doesn't the lower constraints.txt file installed the lower bounds for each library? This seems like a testing gap we'd want to address

[daniel-sanche]

I'm finding all these wrappers very hard to follow. Is there any way we can filter these out instead of wrapping, like I suggested in the other PR?

If we do add extra helpers, I don't think we should need these three, along with additional _prep_wrapped_messages and _wrap_method implementations in the async transport. Can we try to simplify the wrapping logic?

[daniel-sanche]

nit: TYPE_CHECKING blocks should probably come at the end of the import list, not the middle

[daniel-sanche]

suggestion: should we apply this one the same way? Or save that for a follow-up?

[daniel-sanche]

It's not really clear to me why we have these two internal pointers. And it seems to make less sense now that we have even more interceptors being applied. Do you know the reason? Should we keep _logged_channel?

[daniel-sanche]

My first thought was that we'd also need an async context manager, but some brief research seems like that's not the case. Did you confirm this?

[daniel-sanche]

suggestion: have you considered making this context manager into a class? There's a lot of complex state and wrapping logic here, and it's a bit difficult to track all the state here.

I have a feeling it could be easier to work with by implementing __enter__ and __exit__ methods instead, and then some helpers like _build_attributes()

Don't feel like you have to re-write it, but something to consider. If you do want to keep the current code, is it possible to at least flatten some of the wrappers, or break the attribute creation into a separate function?

[daniel-sanche]

Context managers have an expectation that they yield only once. It looks like this could be called after the span was already yielded