feat: remove product-config & clean up by razvan · Pull Request #692 · stackabletech/spark-k8s-operator

razvan · 2026-06-05T13:04:44Z

merging

razvan · 2026-06-10T10:21:01Z

--- PASS: kuttl (1336.26s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (136.48s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (137.32s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.0.1 (278.24s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (130.02s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (200.23s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_spark-connect-4.1.1_s3-use-tls-true (137.33s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (322.81s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (66.59s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (192.76s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (165.33s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (175.59s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.0.1 (64.51s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (61.82s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (116.69s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (141.93s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (192.06s)
PASS

…ct-config

…oleGroupRef

…lper

…r JVM args

…M args

…+ ValidatedRoleConfig

…fig (+ ValidatedRoleConfig for connect)

… document the retained raw CR

…op redundant params

adwk67 · 2026-06-17T11:43:35Z

--- FAIL: kuttl (3221.28s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (434.10s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (531.54s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (315.27s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (315.65s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (809.44s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (88.17s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (603.49s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (658.15s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (180.93s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (194.22s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.1.1_iceberg-latest-1.11.0 (98.75s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (151.20s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (230.70s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (166.80s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (174.24s)
        --- FAIL: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (974.16s)
FAIL

Re-run failed:

--- PASS: kuttl (304.70s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (304.68s)
PASS

adwk67 · 2026-06-23T12:34:06Z

Following op-rs bump:

--- PASS: kuttl (1497.44s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (57.88s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (121.61s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (295.98s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (140.07s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (90.13s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (308.06s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.4.2_zookeeper-latest-3.9.5 (425.17s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (127.30s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (126.02s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.1.1_iceberg-latest-1.11.0 (57.50s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (178.69s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (144.74s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (340.62s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (35.93s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (137.35s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (326.84s)
PASS

…ogging test

maltesander · 2026-06-24T14:10:55Z

--- FAIL: kuttl (2612.82s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.8_s3-use-tls-true (122.35s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-false (202.58s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-4.1.1 (102.14s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-3.5.8_ny-tlc-report-0.3.0 (265.29s)
        --- PASS: kuttl/harness/logging_openshift-false_spark-logging-4.1.1_ny-tlc-report-0.3.0 (287.99s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-false (147.39s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-4.1.1_s3-use-tls-true (150.43s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.8_s3-use-tls-false (136.17s)
        --- PASS: kuttl/harness/spark-history-server_openshift-false_spark-3.5.8_s3-use-tls-true (151.52s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-3.5.8_iceberg-latest-1.11.0 (46.32s)
        --- PASS: kuttl/harness/iceberg_openshift-false_spark-iceberg-4.1.1_iceberg-latest-1.11.0 (60.32s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3_openshift-false_spark-3.5.8 (100.09s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-false (118.03s)
        --- FAIL: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (967.57s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-4.1.1_s3-use-tls-true (122.84s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-3.5.8_hdfs-latest-3.5.0_zookeeper-latest-3.9.5 (164.40s)
        --- PASS: kuttl/harness/custom-log-directory_openshift-false_spark-4.1.1_hdfs-latest-3.5.0_zookeeper-latest-3.9.5 (187.27s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-3.5.8 (36.41s)
        --- PASS: kuttl/harness/spark-examples_openshift-false_spark-4.1.1 (42.44s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-3.5.8_s3-use-tls-false (120.21s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.8_s3-use-tls-false (135.84s)
        --- PASS: kuttl/harness/hbase-connector_openshift-false_spark-hbase-connector-3.5.8_hbase-2.6.4_hdfs-latest-3.5.0_zookeeper-latest-3.9.5 (266.73s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-true (138.03s)
        --- PASS: kuttl/harness/spark-ny-public-s3_openshift-false_spark-3.5.8_s3-use-tls-true (108.38s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-3.5.8 (94.90s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-4.1.1_ny-tlc-report-0.3.0 (107.25s)
        --- PASS: kuttl/harness/pyspark-ny-public-s3-image_openshift-false_spark-3.5.8_ny-tlc-report-0.3.0 (104.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-3.5.8 (182.37s)
        --- PASS: kuttl/harness/product-config-compat_openshift-false_spark-regression-3.5.8 (36.06s)
        --- PASS: kuttl/harness/spark-pi-private-s3_openshift-false_spark-4.1.1 (98.20s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-true (207.97s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-3.5.8 (50.55s)
        --- PASS: kuttl/harness/resources_openshift-false_spark-4.1.1 (67.31s)
        --- PASS: kuttl/harness/smoke_openshift-false_spark-4.1.1_s3-use-tls-false (121.76s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-4.1.1 (105.07s)
        --- PASS: kuttl/harness/overrides_openshift-false_spark-3.5.8 (101.38s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-3.5.8_s3-use-tls-true (213.09s)
        --- PASS: kuttl/harness/spark-connect_openshift-false_iceberg-latest-1.11.0_hive-iceberg-4.0.0_spark-connect-4.1.1_s3-use-tls-false (208.50s)
FAIL
ERROR:root:kuttl failed

--- PASS: kuttl (221.96s)
    --- PASS: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/delta-lake_openshift-false_spark-delta-lake-4.1.1 (221.95s)
PASS

razvan added 2 commits June 5, 2026 14:55

feat: remove product-config & clean up

517fe43

update op-rs patch

dfa3307

razvan mentioned this pull request Jun 5, 2026

tracking: Smooth Operator Iteration 2: Remove product-config stackabletech/issues#851

Open

16 tasks

razvan added 3 commits June 5, 2026 17:04

add regression test

af9492a

fix merging properties for the history controller

b6b1452

update changelog

f210846

maltesander self-requested a review June 8, 2026 07:43

maltesander reviewed Jun 8, 2026

View reviewed changes

razvan added 2 commits June 8, 2026 11:04

implement review feedback

15f0295

update pull policy in some tests

3e009b8

maltesander reviewed Jun 8, 2026

View reviewed changes

Comment thread rust/operator-binary/src/crd/history.rs Outdated

razvan added 5 commits June 8, 2026 16:08

remove shs argument from build_configmap()

d75ec31

Merge branch 'main' into refactor/remove-product-config

f283256

remove Merge impl for history ConfigOverrides.

76931d7

update tests

9bba56a

fix tests and config overrides

1f64bf2

merging

razvan marked this pull request as ready for review June 9, 2026 11:56

maltesander reviewed Jun 9, 2026

View reviewed changes

Comment thread rust/operator-binary/src/crd/history.rs Outdated

Comment thread rust/operator-binary/src/spark_k8s_controller.rs Outdated

Comment thread rust/operator-binary/src/history/controller/validate.rs Outdated

Comment thread rust/operator-binary/src/history/controller/validate.rs Outdated

razvan added 5 commits June 9, 2026 17:25

more cleanups

a24fc24

change namespace type

b410525

replace owner reference with name and uid

791044e

validated history: add metadata and impl Resource

70c7d78

connect validated: make it in line with the history structure

10efca5

maltesander and others added 6 commits June 11, 2026 15:00

fix: use v2 get name/namespace/uid helpers.

8fa3170

fix: cleanup product-config left overs & remove allow dead_code

59411fa

bump op-rs

ffd814e

Option<KeyValueConfigOverrides> -> KeyValueConfigOverrides

77a6aa5

thread metadata through via validated object

ad14fb1

Merge remote-tracking branch 'origin/main' into refactor/remove-produ…

2ca90b8

…ct-config

adwk67 added 11 commits June 16, 2026 15:02

refactor: adopt static vector.yaml and v2 product_logging, removing R…

c9b2bec

…oleGroupRef

refactor(history): build child ObjectMeta via a shared object_meta he…

3620c6f

…lper

refactor(connect): build child ObjectMeta via a shared object_meta he…

f1caec6

…lper

refactor(job): build child ObjectMeta via a shared object_meta helper

3257652

refactor(connect): use v2 role_utils JavaCommonConfig and apply_to fo…

2df3717

…r JVM args

refactor(job): use v2 role_utils JavaCommonConfig and apply_to for JV…

d18e370

…M args

refactor(history): group validated state into ValidatedClusterConfig …

7f62dbf

…+ ValidatedRoleConfig

refactor(connect,job): group validated state into ValidatedClusterCon…

04b8ab6

…fig (+ ValidatedRoleConfig for connect)

refactor(job): thread ValidatedSparkApplication into init_containers;…

08fc433

… document the retained raw CR

refactor: move resource builders into controller/build modules and dr…

8ec3bf2

…op redundant params

fix: test should not assume bash-isms

5d19467

adwk67 and others added 16 commits June 23, 2026 14:36

bump op-rs to 0.112

e372c1d

Merge branch 'main' into refactor/remove-product-config

e847741

fix: use v2 prometheus annotations and label helper

8430d4b

refactor: use v2 container builder

59246b3

refactor: consolidate logging, remove errors

fe720a8

refactor: switch to v2 ListenerClassName

5ff6b43

refactor: switch vector config map name to ConfigMapName v2 type

1569570

refactor: switch internal constants to VolumeName

e39a874

refactor: validate config overrides upfront

60b0ad1

refactor: remove redundant parameters from Validated* structs

53829d8

refactor: consolidate logging for connect

71d04c5

docs: consolidate upstream / v2 comments

744fd18

chore: bump tokio 1.52, clap 4.6

b7694a6

fix: remove vector file tests

07cd672

fix: consolidate vector logging, adapt snapshot tests, add executor l…

4d1a2ed

…ogging test

Merge branch 'main' into refactor/remove-product-config

e6001c0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat: remove product-config & clean up#692

feat: remove product-config & clean up#692
razvan wants to merge 69 commits into
mainfrom
refactor/remove-product-config

razvan commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Jun 10, 2026

Uh oh!

adwk67 commented Jun 17, 2026

Uh oh!

adwk67 commented Jun 23, 2026

Uh oh!

maltesander commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

razvan commented Jun 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

razvan commented Jun 10, 2026

Uh oh!

adwk67 commented Jun 17, 2026

Uh oh!

adwk67 commented Jun 23, 2026

Uh oh!

maltesander commented Jun 24, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants