feat: add queryMetadata config hook with BigQuery job-label support#11065
Open
darrenjl wants to merge 2 commits into
Open
feat: add queryMetadata config hook with BigQuery job-label support#11065darrenjl wants to merge 2 commits into
darrenjl wants to merge 2 commits into
Conversation
…etadata (e.g. BigQuery job labels) to every data-source query for cost attribution and trace correlation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #11054, references #7753.
Adds a generic
queryMetadataconfiguration hook that Cube evaluates once per REST/GraphQL request and threads through to the driver asoptions.queryMetadata. BigQuery is the first consumer: it maps the entries to BigQuery job labels, which surface inINFORMATION_SCHEMA.JOBSfor cost attribution and trace correlation.Usage
Set
x-request-idin your upstream app or AI agent; Cube already uses that header verbatim asrequestId(withtraceparentas fallback), so it flows straight through to the label. Query cost by session:Why a generic hook rather than driver-specific options
Two requirements can't be satisfied by static driver config:
requestIdis dynamic per request — it can't live indriverFactoryconfig set at startup.securityContextdoesn't reach the driver'squery()today — the only workaround isdriverFactory(context)building a fresh driver per tenant, which forcescontextToOrchestratorIdand makes per-request metadata impossible.A hook evaluated in
contextByReq— where bothrequestIdandsecurityContextare in scope — solves both and serves every driver without further core changes.Changes
New config hook (cross-package)
queryMetadata?: (context: RequestContext) => Record<string,string> | Promise<...>added toCreateOptions(server-core),ApiGatewayOptions(api-gateway), validated withJoi.func()ApiGateway.contextByReqalongsideextendContextqueryMetadataonQueryBody→CacheQueryResultOptions→queryWithRetryAndRelease→ the_queryobject passed toclient.query(req.query, req.values, req), mirroringrequestIdexactlyQueryOptionsin@cubejs-backend/base-drivergains an explicitqueryMetadata?fieldBigQuery driver
buildJobLabels(options)readsoptions.queryMetadataand sanitizes values to BigQuery label constraints (lowercase,[a-z0-9_-]only, max 63 chars)runQueryJob()— the single chokepoint forquery()andloadPreAggregationIntoTable()stream()unchanged (out of scope for this PR)Tests
packages/cubejs-bigquery-driver/test/unit/labels.test.ts— 8 cases, mocked@google-cloud/bigquery, no GCP credentials requiredDocs
docs-mintlify/reference/configuration/config.mdx— newquery_metadatasectiondocs-mintlify/admin/connect-to-data/data-sources/google-bigquery.mdx— updated Job Labels sectionKnown limitations (future PRs)
contextByReq). Refresh-worker pre-aggregation builds and the SQL API are not request-scoped and won't carry metadata — semantically correct (no upstream request).stream()is not labelled this PR.QUERY_TAG, SQL comments, etc.) can adoptoptions.queryMetadatawithout further core changes.Checklist