[spark] Support custom Paimon lake table paths

[BackendArchitectX]

Purpose

Linked issue: close #3832

Support Spark reads from Paimon lake tables whose database and/or table name is configured independently from the Fluss table path.

Previously, Spark rejected lake reads when the resolved Paimon table path differed from the Fluss table path.

Brief change log

• Resolve the configured lake table path before creating the Spark lake source.
• Use the resolved Paimon path for lake reads while preserving the original Fluss table path.
• Enable lake predicate pushdown for tables using custom lake paths.
• Coverage includes:
  • custom database mapping
  • custom table mapping
  • combined database and table mapping
  • lake-only reads
  • lake + log union reads
  • predicate pushdown with a custom lake path
  • preservation of the existing no-lake-snapshot fallback

Tests

Local validation (Linux / JDK 17):

• SparkLakePaimonLogTableReadTest
  • 18 tests passed
• SparkLakePaimonPrimaryKeyTableReadTest
  • 10 tests passed
• spotless:check for fluss-spark-common and fluss-spark-ut
• checkstyle:check for fluss-spark-common and fluss-spark-ut
• git diff --check

API and Format

No public API or storage format changes.

Documentation

No documentation changes are required. This change enables Spark support for the existing custom Paimon lake table path configuration.

[BackendArchitectX]

Hi @luoyuxia @YannByron, PTAL when you have a chance.

This is the scoped Spark follow-up to #3476 tracked by #3832. The production change is limited to resolving the configured Paimon table path when creating the lake source and removing the obsolete custom-path predicate-pushdown guard.

I rebased onto the latest main and reran the relevant Paimon Spark suites locally on Linux/JDK 17:

SparkLakePaimonLogTableReadTest: 18/18 passed
SparkLakePaimonPrimaryKeyTableReadTest: 10/10 passed

CI is currently waiting for maintainer approval to run.

Thanks!