improvement(datamodel): server-side column tag & glossary filter for Dashboard Data Models - #29852
improvement(datamodel): server-side column tag & glossary filter for Dashboard Data Models#29852sonika-shah wants to merge 3 commits into
Conversation
❌ PR checklist incompleteThis PR cannot be merged until the following are addressed on its linked issue:
The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically. Maintainers can bypass this check by adding the |
github-actions
Bot
added
backend
safe to test
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
This PR brings Dashboard Data Model column search/tag/glossary filtering to parity with Tables by moving the tree-pruning + tag-resolution logic into a shared backend utility and wiring new query params through the Dashboard Data Model /columns/search endpoints, with the UI updated to use server-side filtering instead of client-side per-page filtering.
Changes:
- Backend: Extract column search + tag/glossary filtering into
ColumnSearchUtiland apply it to both Table and Dashboard Data Model column search paths. - Backend API: Add
tags,glossaryTerms,sortBy, andsortOrderquery params to Dashboard Data Model/columns/searchendpoints (by-id and by-fqn). - Frontend: Update Data Model “Model” tab to use server-side tag/glossary filtering and auto-expand ancestors of matches; add integration coverage for cross-page tag filtering.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| openmetadata-ui/src/main/resources/ui/src/rest/dataModelsAPI.ts | Adds tags/glossaryTerms params to Data Model column search request typing. |
| openmetadata-ui/src/main/resources/ui/src/components/Dashboard/DataModel/DataModels/ModelTab/ModelTab.component.tsx | Switches column tag/glossary filtering to server-side search and auto-expands match ancestors. |
| openmetadata-service/src/main/java/org/openmetadata/service/util/ColumnSearchUtil.java | New shared util for column-tree pruning, sorting, flattening, and tag resolution for filtering. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/datamodels/DashboardDataModelResource.java | Adds tag/glossary + sort query params to both Data Model column search endpoints and parses them. |
| openmetadata-service/src/main/java/org/openmetadata/service/resources/databases/TableResource.java | Updates ColumnTagFilter import to the shared util type. |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/TableRepository.java | Delegates column pruning/tag filtering logic to ColumnSearchUtil (no intended behavior change). |
| openmetadata-service/src/main/java/org/openmetadata/service/jdbi3/DashboardDataModelRepository.java | Replaces flat search with server-side tree pruning + tag-resolution across all pages (on deep-copied tree). |
| openmetadata-integration-tests/src/test/java/org/openmetadata/it/tests/DashboardDataModelResourceIT.java | Adds integration test asserting tag filtering finds matches beyond the first page (by-fqn and by-id). |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
d406834 to
134b4fc
Compare
Jest test CoverageUI tests summary
|
…Dashboard Data Models Dashboard Data Models paginate their columns server-side (PAGE_SIZE_LARGE) but the Tags/Glossary column filter ran client-side over the loaded page only, so a tag applied to a column on a later page could not be found — the same cross-page bug that #25063 fixed for Tables (#29324), which never covered data models. Backend: - Extract the shared column search + tag/glossary filtering from TableRepository into ColumnSearchUtil (pruneColumnsToMatches, columnComparator, resolveColumnTagsForFilter, ColumnTagFilter). Both Table and DashboardDataModel operate on the same Column tree. - Add tags/glossaryTerms (plus sortBy/sortOrder) to both DashboardDataModel /columns/search endpoints and resolve direct + glossary-derived tags server-side, pruning the tree to matched ancestor paths across all pages. - Data model text search now returns the nested tree (matched column under its ancestors) instead of a flat list, consistent with Tables. Search matches name/displayName only, matching the Table endpoint. Frontend (ModelTab): - Replace the client-side useTreeTagFilter with a server-side filter that calls searchDataModelColumnsByFQN with tags/glossaryTerms, resets to page 1 on change, sources filter options from the full entity columns, and auto-expands ancestors of matches. Tests: - Integration test asserting the tag filter finds a column tagged on a later page (by-fqn and by-id).
134b4fc to
721f7ec
Compare
|
Code Review 🚫 Blocked 1 resolved / 3 findingsImplements server-side column filtering for Dashboard Data Models using a new shared utility, ensuring consistency with existing Table schemas. Findings include redundant CSV parsing logic in resources and an accidental commit of local configuration settings that must be resolved. 🚨 Bug: Accidental local-dev DB/search config committed to openmetadata.yaml📄 conf/openmetadata.yaml:329 📄 conf/openmetadata.yaml:334 📄 conf/openmetadata.yaml:557 The delta commit changes
These change the out-of-the-box behavior for every consumer of the default config and are almost certainly an accidental commit of a local environment. Revert Revert the unrelated openmetadata.yaml default changes (driverClass, JDBC url/host, searchType) back to their original values; drop the hardcoded 192.168.29.109 host.💡 Quality: Duplicated tag-filter CSV parsing in two resources📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/datamodels/DashboardDataModelResource.java:1018-1032 📄 openmetadata-service/src/main/java/org/openmetadata/service/resources/databases/TableResource.java:2203-2217
Centralize CSV parsing so both resources delegate to ColumnSearchUtil.✅ 1 resolved✅ Bug: getExpandAllKeysToDepth imported from wrong module
🤖 Prompt for agentsOptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
openmetadata-ui/src/main/resources/ui/src/components/Dashboard/DataModel/DataModels/ModelTab/ModelTab.component.tsx:517
_expandedRowKeysis updated (deep-link + auto-expand effect), but the table isn't controlled withexpandedRowKeys/onExpand. As a result, deep links and the new auto-expansion for search/tag filters won't actually expand ancestors, which can hide nested matches.
expandable={{
...getTableExpandableConfig<Column>(false, 'text-link-color'),
rowExpandable: (record) => !isEmpty(record.children),
}}
- ColumnSearchUtil: match description in column text search (restores the data-model behavior and aligns Table with its documented search) and add ColumnTagFilter.fromCsv so both resources share one CSV parser - TableResource/DashboardDataModelResource: delegate to ColumnTagFilter.fromCsv, drop the duplicated parseColumnTagFilters/parseFqnCsv helpers - ModelTab: guard the table onChange to filter actions only so client-side sort no longer resets pagination
| private static boolean columnTextMatches(Column column, String searchTerm) { | ||
| boolean nameMatches = | ||
| column.getName() != null && column.getName().toLowerCase().contains(searchTerm); | ||
| boolean displayNameMatches = | ||
| column.getDisplayName() != null | ||
| && column.getDisplayName().toLowerCase().contains(searchTerm); | ||
| boolean descriptionMatches = | ||
| column.getDescription() != null | ||
| && column.getDescription().toLowerCase().contains(searchTerm); | ||
| return nameMatches || displayNameMatches || descriptionMatches; | ||
| } |
| const tagFilter = useMemo(() => { | ||
| const tags = getAllTags(data ?? []); | ||
| const tags = getAllTags([...(dataModel?.columns ?? []), ...(data ?? [])]); | ||
|
|
||
| return groupBy(uniqBy(tags, 'value'), (tag) => tag.source) as Record< |
| boolean descriptionMatches = | ||
| column.getDescription() != null | ||
| && column.getDescription().toLowerCase().contains(searchTerm); | ||
| return nameMatches || displayNameMatches || descriptionMatches; |
Code Review ✅ Approved 3 resolved / 3 findingsImplements server-side column tag and glossary filtering for Dashboard Data Models, addressing the cross-page search bug and consolidating logic into a shared utility. No issues were found. ✅ 3 resolved✅ Bug: getExpandAllKeysToDepth imported from wrong module
✅ Bug: Accidental local-dev DB/search config committed to openmetadata.yaml
✅ Quality: Duplicated tag-filter CSV parsing in two resources
OptionsDisplay: compact → Showing less information. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
|
|
🔴 Playwright Results — 1 failure(s), 27 flaky✅ 4527 passed · ❌ 1 failed · 🟡 27 flaky · ⏭️ 38 skipped
Genuine Failures (failed on all attempts)❌
|
Related to #25063
#25063 added a server-side column tag/glossary filter to the Table schema view (implemented in #29324), fixing a cross-page bug where a tag applied to a column on a later page could not be found because filtering ran client-side over the loaded page only.
Dashboard Data Models have the identical setup — columns paginate server-side (
PAGE_SIZE_LARGE), the schema table exposes Tags/Glossary column filters — but their/columns/searchendpoint never got the filter, so data models still exhibit the original bug. This PR brings them to parity and extracts the logic into a shared util so the two entities stay consistent.Why data models specifically
Table and Dashboard Data Model are the only two entities whose children are the
Columntype and fetched page-by-page from a/columns/searchendpoint. Every other entity with a column/field tag filter (Container, Topic, SearchIndex, API Endpoint, File, Worksheet, Dashboard, Pipeline) loads all children inline, so its client-side filter already spans everything — no server-side change needed.Backend
ColumnSearchUtil— the column search + tag/glossary filtering (pruneColumnsToMatches,columnComparator,resolveColumnTagsForFilter,ColumnTagFilter) is extracted fromTableRepositoryinto a shared util.TableRepositorynow delegates to it (no behavior change for Tables).DashboardDataModelResource— both/columns/searchendpoints gaintags,glossaryTerms,sortBy,sortOrderparams.DashboardDataModelRepository— replaces the old flat search with server-side tree pruning + direct-and-derived tag resolution across all pages, on a deep copy of the column tree.Frontend (
ModelTab)useTreeTagFilterwith a server-side filter that callssearchDataModelColumnsByFQNwithtags/glossaryTerms, resets to page 1 on change, sources filter options from the full entity columns, and auto-expands ancestors of matches — mirroringSchemaTable.Behavior notes
name/displayNameonly (the old data-model search also matcheddescription); this aligns it with the Table endpoint.Tests
mvn install(service + integration-tests) passes; frontendtsc --noEmitclean.Greptile Summary
This PR adds server-side column tag and glossary filtering for Dashboard Data Models. The main changes are:
ColumnSearchUtil./columns/searchsupport for tag, glossary, and sort parameters.Confidence Score: 5/5
This looks safe to merge.
Important Files Changed
Reviews (4): Last reviewed commit: "review: expose sortBy/sortOrder on Searc..." | Re-trigger Greptile
Context used (3)