fix(table): server-side column tag filter across all pages (#25063)#29324
Conversation
Column tag filtering ran client-side on the loaded page only, so columns matching a tag on a later page stayed hidden once columns paginate server-side. Push the filter down to the same layer as pagination. Backend: add tags/glossaryTerms query params to the existing /columns/search endpoints (matching the /columns/grid vocabulary) and filter the full matching column set before sort/pagination in searchTableColumnsInternal. Matching uses a single tag_usage prefix lookup (getTagsByPrefix) instead of hydrating tags onto every column. Frontend: route the SchemaTable tag/glossary dropdown through the search call, reset to page 1 on change, drop the client-side onFilter, and source the dropdown options from the full table columns so every tag is selectable regardless of page.
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR fixes table schema column tag/glossary filtering so it works across the full column set with server-side pagination (instead of only filtering the currently loaded page on the client).
Changes:
- Backend: adds
tagsandglossaryTermsquery params to/columns/searchendpoints and applies tag-based filtering before sort/pagination inTableRepository. - Frontend: routes tag/glossary filters through the
/columns/searchcall, removes client-side filtering, and sources filter options from the fulltable.columns. - Tests: adds an integration test to validate tag filtering across pages and updates SchemaTable unit tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-ui/src/main/resources/ui/src/rest/tableAPI.ts | Extends search params to include tags/glossaryTerms for the table columns search API. |
| openmetadata-ui/src/main/resources/ui/src/components/Database/SchemaTable/SchemaTable.component.tsx | Converts tag/glossary filtering to server-side by driving it through search + controlled filter state. |
| openmetadata-ui/src/main/resources/ui/src/components/Database/SchemaTable/SchemaTable.test.tsx | Adds coverage ensuring tag filter options come from full table columns, not just the page slice. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/databases/TableResource.java | Adds tags/glossaryTerms query params and parses them into a filter set for repository search. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java | Applies tag/glossary filtering to matched columns before sort/pagination using a prefix tag-usage lookup. |
| openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/TableResourceIT.java | Adds an IT validating tag filtering returns matches across pages (including when the match is beyond page 1). |
…filter The shorter searchTableColumns* overloads still called the new arity without the filterTagFQNs argument (openmetadata-service compile failure), and the IT used the wrong TagSource package. Pass null through the delegators and import the nested TagLabel.TagSource.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Jest test CoverageUI tests summary
|
🟡 Playwright Results — all passed (38 flaky)✅ 4448 passed · ❌ 0 failed · 🟡 38 flaky · ⏭️ 37 skipped
🟡 38 flaky test(s) (passed on retry)
How to debug locally# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip # view trace |
…ation tags) filterColumnsByTags resolved column tags via getTagsByPrefix, which returns only direct tag_usage rows and strips certification-classification tags. The column tag filter dropdown is built from column.tags (direct + glossary-derived tags), so selecting a derived or certification tag returned zero matches. Resolve column tags the same way the column list responses do: raw tag_usage prefix query (no certification stripping) enriched with glossary-derived tags via batchFetchDerivedTags. Derived-tag fetch failures degrade to direct tags. Add an IT covering the glossaryTerms param, the derived classification tag, and the by-id /columns/search endpoint.
…filter-server-side # Conflicts: # openmetadata-ui/src/main/resources/ui/src/components/Database/SchemaTable/SchemaTable.component.tsx
2029ab7 to
d971064
Compare
…filter-server-side
|
@gitar-bot re: the The flagged line belongs to a different PR: The orphan concern is also already handled on deleteOrUpdateChildren(entity, indexMapping);
if (Entity.TABLE.equals(entityType)) {
deleteTableColumns((Table) entity);
} else {
deleteDescendantColumns(entity, entityType); // #29549: clears TABLE_COLUMN docs for db/schema/service
}So a recursive hard-delete of a database/schema/service still clears descendant column docs via Not actioning this in this PR (out of scope — belongs to #29322/#29549). If there's a residual gap, it should be raised against those PRs, not this one. |
Acknowledged — the feedback is well-reasoned and the attribution is correct. The flagged line ( The finding will be dismissed as out-of-scope for this PR. If a residual gap exists, it belongs against #29322 or #29549. |
|
|
Code Review ✅ Approved 4 resolved / 4 findingsMoves column tag and glossary filtering to the backend to ensure accurate cross-page results and proper tree structure. Resolved issues regarding derived tag inheritance, filter-triggered pagination resets, and unintended tree mutation. ✅ 4 resolved✅ Edge Case: Tag filter may miss columns whose tag is derived/inherited
✅ Quality: onChange resets to page 1 on any antd table change, not just filters
✅ Quality: pruneColumnsToMatches mutates the loaded table's column tree in place
✅ Bug: Table column search docs may orphan on ancestor cascade delete
OptionsDisplay: compact → Showing less information. Comment with these commands to change:
Was this helpful? React with 👍 / 👎 | Gitar |
Fixes #25063
Screen.Recording.2026-06-29.at.1.05.17.PM.mov
Problem
What changed
Backend
tags+glossaryTermsquery params (comma-separated FQNs) to the two/columns/searchendpoints (by-id and by-fqn). All other column endpoints are untouched.totalnow counts top-level columns that contain a match — the correct unit once rows are nested.tagsandglossaryTermsgroups are AND-ed (matches the table's multi-column filter behavior).Frontend (Schema table)
Tests
Greptile Summary
This PR fixes server-side column tag filtering in the table Schema view. Previously, tag filters applied client-side over only the currently loaded page, so columns with matching tags on other pages were invisible. Now both
/columns/searchendpoints accepttagsandglossaryTermsquery params, the backend prunes the full column tree server-side (keeping matches at their real nested depth), and pagination counts top-level root columns rather than flattened matches.ColumnTagFilterrecord, deep-copies the column tree before pruning, resolves tags directly via the raw DAO (avoiding the certification-classification strip in the service layer), and wiresparseColumnTagFiltersinto both by-id and by-fqn search endpoints.useTreeTagFilterclient-side hook, stores active filter state inactiveTagFilter, delegates filtering tosearchTableColumnson change (resetting to page 1), and sources tag filter dropdown options from the fulltable.columnsentity rather than only the loaded page — with ancestors auto-expanded to reveal deep matches.Confidence Score: 5/5
Safe to merge. The server-side filter runs on a deep copy of the column tree, pagination and field hydration apply to the correct page slice, and the frontend correctly resets pagination and switches between fetch paths on filter change.
The core tree-pruning logic correctly deep-copies before mutation, deduplicates tag filter options by FQN, and wires PIIMasker against the page slice whose objects are shared with the returned roots. Integration tests cover cross-page, derived-tag, AND-between-groups, and nested-STRUCT pruning scenarios. No functional regressions were found in the changed code paths.
No files require special attention. The one inline comment is a documentation nit around the PIIMasker call in TableRepository.
Important Files Changed
Sequence Diagram
%%{init: {'theme': 'neutral'}}%% sequenceDiagram participant UI as SchemaTable (Frontend) participant API as TableResource (Backend) participant Repo as TableRepository participant DAO as TagUsageDAO UI->>API: "GET /columns/search?tags=PII.Sensitive&glossaryTerms=..." API->>API: parseColumnTagFilters(tags, glossaryTerms) API->>Repo: searchTableColumnsByFQN(..., columnTagFilter) Repo->>Repo: deepCopyList(table.columns) Repo->>DAO: getTagsByPrefix(tableFQN, ".%") DAO-->>Repo: directTagsByHash (raw, includes certification tags) Repo->>Repo: batchFetchDerivedTags(allDirectTags) Repo->>Repo: addDerivedTagsWithPreFetched → effectiveTagsByHash Repo->>Repo: pruneColumnsToMatches(allColumns, searchTerm, filter, tagsByHash) Note over Repo: Tree pruned: matched nodes kept at depth,<br/>non-matching siblings/children removed Repo->>Repo: sort root-level columns, paginate Repo->>Repo: populateEntityFieldTags(paginatedColumns) Repo-->>API: "ResultList(paginatedRoots, total=rootsWithMatch)" API-->>UI: "TableColumnList {data, paging}" UI->>UI: setTableColumns(pruneEmptyChildren(data)) UI->>UI: "auto-expand ancestors of matches (depth=MAX_SAFE_INTEGER)" UI->>UI: tagFilter options from getAllTags([table.columns, tableColumns])%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%% sequenceDiagram participant UI as SchemaTable (Frontend) participant API as TableResource (Backend) participant Repo as TableRepository participant DAO as TagUsageDAO UI->>API: "GET /columns/search?tags=PII.Sensitive&glossaryTerms=..." API->>API: parseColumnTagFilters(tags, glossaryTerms) API->>Repo: searchTableColumnsByFQN(..., columnTagFilter) Repo->>Repo: deepCopyList(table.columns) Repo->>DAO: getTagsByPrefix(tableFQN, ".%") DAO-->>Repo: directTagsByHash (raw, includes certification tags) Repo->>Repo: batchFetchDerivedTags(allDirectTags) Repo->>Repo: addDerivedTagsWithPreFetched → effectiveTagsByHash Repo->>Repo: pruneColumnsToMatches(allColumns, searchTerm, filter, tagsByHash) Note over Repo: Tree pruned: matched nodes kept at depth,<br/>non-matching siblings/children removed Repo->>Repo: sort root-level columns, paginate Repo->>Repo: populateEntityFieldTags(paginatedColumns) Repo-->>API: "ResultList(paginatedRoots, total=rootsWithMatch)" API-->>UI: "TableColumnList {data, paging}" UI->>UI: setTableColumns(pruneEmptyChildren(data)) UI->>UI: "auto-expand ancestors of matches (depth=MAX_SAFE_INTEGER)" UI->>UI: tagFilter options from getAllTags([table.columns, tableColumns])Comments Outside Diff (1)
openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java, line 3403-3409 (link)When
fieldsincludes"profile",setColumnProfileandpopulateEntityFieldTagsare called onmatchingColumns(the full tag-filtered, sorted set) rather than onpaginatedResults. For a table with many columns and an active tag filter that still returns dozens of matches, this fetches profile data for every matching column even though only a page of them is returned to the caller. This is pre-existing behaviour in the non-tag path, but the tag-filter path now makes it more reachable with large result sets.Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Reviews (13): Last reviewed commit: "Merge branch 'main' into fix/25063-colum..." | Re-trigger Greptile