perf: cache dictionary arc pointer - #24418
Conversation
|
cc @zhuqi-lucas since this was a follow up you mentioned 😄 |
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #24418 +/- ##
==========================================
- Coverage 81.23% 81.21% -0.02%
==========================================
Files 1111 1113 +2
Lines 390208 392265 +2057
Branches 390208 392265 +2057
==========================================
+ Hits 316990 318591 +1601
- Misses 54591 54924 +333
- Partials 18627 18750 +123 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
CI failures are not related |
alamb
left a comment
There was a problem hiding this comment.
Thank you @Rich-T-kid -- this looks quite clever and can probably generate an observable speedup for end to end benchmarks that read data as parquet dictionaries. Did you try that?
Also, I wonder if it would be easier to test some of these dictionary related improvements if you added a config setting to treat string columns in parquet files as dictionary (rather than Stringview) 🤔
@alamb no not exactly. I have made some made tweaks locally to the existing tpch benchmarks by changing datatypes to dict and they've shown improvement. I have this PR #24227 that implements your suggestion #24111 (comment) , it would be nice if we could run the tpch benchmarks on it while having the flag enabled. I think we could see some very nice results. Ill tweak this adjust this PR |
6998281 to
d22b079
Compare
|
once #24416 we can run the benchmarks on this PR, locally i saw up to a 50% boost for the high cardinality branch |
Which issue does this PR close?
Rationale for this change
append_val is called once per new group on the scalar (streaming) code path. Previously it hashed a single-element slice of the dictionary values array on every call, paying create_hashes fixed overhead 65k times for a high-cardinality batch. Caching the full values array hash keyed on Arc::ptr_eq collapses that to one vectorized hash pass per batch, yielding a 2× speedup on the all-unique case with no measurable regression elsewhere.
What changes are included in this PR?
Are these changes tested?
nothing logically changed.
Are there any user-facing changes?
no