Skip to content

fix(viewtools): normalize query in raw() search path (#36026)#36170

Merged
fabrizzio-dotCMS merged 4 commits into
mainfrom
issue-36026-raw-aggregation-field-normalization
Jun 16, 2026
Merged

fix(viewtools): normalize query in raw() search path (#36026)#36170
fabrizzio-dotCMS merged 4 commits into
mainfrom
issue-36026-raw-aggregation-field-normalization

Conversation

@fabrizzio-dotCMS

@fabrizzio-dotCMS fabrizzio-dotCMS commented Jun 15, 2026

Copy link
Copy Markdown
Member

Proposed Changes

Follow-up fix for #36026 (the merged change was marked QA : Failed).

Problem

The Velocity $estool.raw(...) / $estool.esRaw(...) search path forwards the query verbatim, while $estool.search(...) / esSearch(...) lowercases the whole query via StringUtils.lowercaseStringExceptMatchingTokens before executing it. As a result, a camelCase aggregation field such as "field":"contentType":

  • worked under search() — folded to the physical lowercase index field contenttype, and
  • returned empty buckets silently under raw() — it aggregated over a field name that does not exist in the index.

This surfaced as "raw() returns empty getBuckets=[] for aggregations". The aggregation mapping itself was never at fault — both paths map the response through the same ContentSearchResponse.from(...).

Root cause (single point)

search/esSearch lowercase the query before delegating to the raw method; the raw method (esSearchRaw / OSSearchAPIImpl.searchRaw) never did.

Fix (minimal)

Apply the same lowercasing the search path already uses, at the entry of both raw paths:

  • enterprise/.../ESSearchAPIImpl.esSearchRaw(String) (ES — covers neutral raw() and deprecated esRaw())
  • opensearch/OSSearchAPIImpl.searchRaw(String) (OS — symmetric)

raw() now reaches full parity with search() for mixed-case field names. One reused helper invocation per path, no new API. The call is idempotent for the search() path, which delegates here after already lowercasing.

Note: as with search(), this folds the entire query (field names and values), so neither path supports case-sensitive exact matches. This keeps raw() and search() consistent.

Files

  • ESSearchAPIImpl.java (enterprise) — lowercase the query in esSearchRaw
  • OSSearchAPIImpl.java — lowercase the query in searchRaw
  • SearchAPI.java, ESContentTool.java — javadoc updated to the consistent contract

How to test

```bash
./mvnw verify -pl :dotcms-integration -Dcoreit.test.skip=false -Dit.test=ContentSearchToolTest
```

  • rawNormalizesQuery_reachingParityWithSearchraw(camelCase) now populates buckets and matches search(camelCase) and raw(lowercase) bucket counts.
  • rawVtl_rendersTreeAndFlatAggregations — drives the customer's camelCase contentType field end-to-end through Velocity.

Checklist

  • Core compiles (BUILD SUCCESS)
  • ES and OS raw paths fixed symmetrically
  • Regression test updated for parity
  • Integration suite green (run ContentSearchToolTest)

Fixes #36026

🤖 Generated with Claude Code

This PR fixes: #36026

@claude

claude Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

Claude finished @fabrizzio-dotCMS's task in 1m 38s —— View job


Rollback Safety Analysis

  • Read rollback-unsafe categories reference
  • Get full PR diff
  • Analyze diff against unsafe categories
  • Apply appropriate label

Result: ✅ Safe To Rollback

After analyzing the diff against all categories in the rollback-unsafe reference, this PR does not match any unsafe category.

Analysis

The PR makes the following changes:

File Change Category Check
ESSearchAPIImpl.java Adds StringUtils.lowercaseStringExceptMatchingTokens() call at entry of esSearchRaw() No DB/ES mapping change — pure runtime behavior
OSSearchAPIImpl.java Same normalization added to searchRaw() Same as above
SearchAPI.java Javadoc updates only No contract change
ESContentTool.java Javadoc updates only No contract change
ContentSearchToolTest.java New integration test + updates existing test Test-only

Category-by-category reasoning:

  • C-1 / C-2 / C-3 / C-4 (DB/ES structural changes): No schema, mapping, or model version changes. No runonce tasks. ✅ Safe

  • H-1 (One-way data migration): No data transformation on stored content. ✅ Safe

  • H-2 through H-7 (Rename, PK, field type, storage, procedure, NOT NULL): None applicable. ✅ Safe

  • H-8 (VTL Viewtool Contract Change): This PR does not change any public method signature on ESContentTool. The raw(String) and search(String) methods retain the same name, parameter types, and return types (ContentSearchResponse / ContentSearchResults). Javadoc is the only diff — no accessible surface changed. ✅ Safe

    The only behavioral change is that raw() now lowercases the query internally before forwarding it — this makes it behaviorally consistent with search() and fixes a bug where mixed-case field names yielded empty results. Since both N and N-1 return the same ContentSearchResponse type with the same accessors, rolling back to N-1 merely reverts to the previous (broken) behavior for mixed-case queries; it does not break any VTL template that worked before N.

  • M-1 through M-4 (column type, bundle format, REST/GraphQL contract, OSGi): No changes to DB columns, bundle XML, REST response shapes, or OSGi interfaces. ✅ Safe

Label applied: AI: Safe To Rollback

The raw() / esRaw() Velocity search path forwarded the query verbatim while
search() / esSearch() lowercased the whole query via
StringUtils.lowercaseStringExceptMatchingTokens. A camelCase aggregation field
such as "contentType" therefore resolved under search() (folded to the physical
lowercase index field "contenttype") but aggregated over a non-existent field
under raw(), returning empty buckets silently — the regression reported against
the aggregation-tree change.

Fix: apply the same lowercasing the search path already uses at the entry of
both raw paths (enterprise ESSearchAPIImpl.esSearchRaw and
OSSearchAPIImpl.searchRaw), so raw() reaches full parity with search() for
mixed-case field names. The call is idempotent for the search() path, which
delegates here after already lowercasing. This is the minimal change: one reused
helper invocation per raw path, no new API.

Tests: ContentSearchToolTest asserts raw(camelCase) reaches parity with search()
(same bucket count), and RAW_VTL now drives the customer's camelCase field
end-to-end through Velocity.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@fabrizzio-dotCMS fabrizzio-dotCMS force-pushed the issue-36026-raw-aggregation-field-normalization branch from 49bcae4 to d9bb3ef Compare June 15, 2026 19:23
@fabrizzio-dotCMS fabrizzio-dotCMS changed the title fix(viewtools): normalize field references in raw() search path (#36026) fix(viewtools): normalize query in raw() search path (#36026) Jun 15, 2026
@dotCMS dotCMS deleted a comment from github-actions Bot Jun 15, 2026
@github-actions

github-actions Bot commented Jun 15, 2026

Copy link
Copy Markdown
Contributor

❌ Codex Review failed — openai.gpt-5.5

The review job failed before producing output. See the run for details.

Run: #27573946549

@wezell wezell left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good. The lowercasing is hard. Glad you found the fix.

@fabrizzio-dotCMS fabrizzio-dotCMS added this pull request to the merge queue Jun 15, 2026
@github-merge-queue github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Jun 15, 2026
@fabrizzio-dotCMS fabrizzio-dotCMS added this pull request to the merge queue Jun 16, 2026
Merged via the queue into main with commit 7f8eee8 Jun 16, 2026
58 of 60 checks passed
@fabrizzio-dotCMS fabrizzio-dotCMS deleted the issue-36026-raw-aggregation-field-normalization branch June 16, 2026 01:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI: Safe To Rollback Area : Backend PR changes Java/Maven backend code

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

Aggregation return-type change breaks existing VTL templates accessing $results.aggregations

3 participants