[VL] Add hash table memory usage metric when enabling spark.gluten.velox.buildHashTableOncePerExecutor.enabled

[JkSelf]

What changes are proposed in this pull request?

Add hash table memory usage metric when enabling spark.gluten.velox.buildHashTableOncePerExecutor.enabled.

How was this patch tested?

local verified

Was this patch authored or co-authored using generative AI tooling?

[Copilot]

Pull request overview

Adds a new Spark SQL metric to report retained hash table memory usage for Velox broadcast hash joins when the “build hash table once per executor” path is used, wiring the value from native Velox hash table allocation tracking through JNI into Spark metrics.

Changes:

• Track hash table retained bytes in native (HashTableBuilder) and expose it via a new JNI/Java API (HashJoinBuilder.getHashTableMemoryUsage).
• Populate the memory usage value during hash table build (including parallel-build subtables) and report it into a new Spark SizeMetric (hashTableMemorySize).
• Thread the new metric through broadcast hash join context/plumbing so it’s recorded when building the executor-side cached hash table.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
cpp/velox/operators/hashjoin/HashTableBuilder.h	Adds storage/accessors for a cached hash table memory usage value.
cpp/velox/jni/VeloxJniWrapper.cc	Captures allocated bytes during native build and adds a JNI entrypoint to query the metric.
cpp/velox/jni/JniHashTable.h	Declares hashTableMemoryUsage(...) native helper.
cpp/velox/jni/JniHashTable.cc	Implements hashTableMemoryUsage(...) accessor (note: needs fallback for deserialized handles).
backends-velox/src/main/scala/org/apache/gluten/execution/VeloxBroadcastBuildSideCache.scala	Updates the new SQL metric when building a cached broadcast hash table.
backends-velox/src/main/scala/org/apache/gluten/execution/HashJoinExecTransformer.scala	Threads the new metric through BroadcastHashJoinContext.
backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxMetricsApi.scala	Registers hashTableMemorySize as a new Spark size metric.
backends-velox/src/main/java/org/apache/gluten/vectorized/HashJoinBuilder.java	Adds the new native method declaration for querying hash table memory usage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

[Copilot]

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

cpp/velox/jni/JniHashTable.cc:186

• hashTableMemoryUsage(builder) currently just returns HashTableBuilder::hashTableMemoryUsage_, but that field is only set by nativeBuild. Hash tables created via deserializeHashTableDirect (and any other path that doesn't call setHashTableMemoryUsage) will therefore report 0, which contradicts the API comment (“retained bytes ...”). Consider deriving the value from builder->hashTable()->allocatedBytes() (or using it as a fallback) so the JNI/Java API works for all hash table handles.

int64_t hashTableMemoryUsage(std::shared_ptr<HashTableBuilder> builder) {
  VELOX_CHECK_NOT_NULL(builder, "Hash table builder cannot be null");
  return builder->hashTableMemoryUsage();
}