iceberg: support Iceberg REST catalogs via iceberg_* options

[adsharma]

Problem

The iceberg extension embeds a DuckDB instance to read Iceberg tables, but exposed no way to configure DuckDB's Iceberg REST catalog connection. As a result:

• REST-catalog data lakes (AWS Glue / SageMaker Lakehouse, S3 Tables, Apache Polaris, Lakekeeper, Nessie, Unity Catalog, ...) were unusable from Ladybug.
• The only resolution path was filesystem-based, which depends on metadata/version-hint.text — a file only produced by HadoopCatalog/HadoopTables. PyIceberg/Java Iceberg writers never generate it, forcing users to run an external program to update the hint file after every streaming commit, with consistency and file-locking risks under concurrent writers.

Context: LadybugDB/ladybug#739 (reply in thread)

Solution

Expose DuckDB's Iceberg REST catalog surface through Ladybug extension options. When configured, the embedded DuckDB instance executes

CREATE SECRET iceberg_rest_secret (TYPE ICEBERG, TOKEN '…', …);
ATTACH 'warehouse' AS iceberg_catalog (TYPE ICEBERG, SECRET …, ENDPOINT '…');

so the current metadata location is resolved through the catalog at query time — no version-hint.text, safe under concurrent commits.

New extension options

Option	Description
iceberg_warehouse	Warehouse identifier; setting it enables REST catalog mode
iceberg_endpoint	REST catalog endpoint URL
iceberg_endpoint_type	e.g. glue, s3_tables
iceberg_authorization_type	sigv4 (AWS Glue)
iceberg_token	OAuth2 bearer token (confidential)
iceberg_client_id / iceberg_client_secret	OAuth2 client credentials (confidential)
iceberg_oauth2_server_uri	OAuth2 token endpoint

Every option can also come from an environment variable (same name, upper-case accepted), keeping credentials out of scripts.

Table name resolution

iceberg_scan / iceberg_metadata / iceberg_snapshots accept fully qualified names: 'iceberg_catalog.namespace.table' (fixed attach alias) or 'namespace.table' (auto-prefixed). Any other 3-part prefix is rejected with a clear error. Filesystem-path mode is unchanged when REST options are not set.

Time travel

For catalog tables, snapshot_from_id / snapshot_from_timestamp map to DuckDB's AT (VERSION => …) / AT (TIMESTAMP => …) clause; the two are mutually exclusive and other scan options are rejected (they don't apply to catalog tables).

Usage

CALL iceberg_warehouse='warehouse';
CALL iceberg_endpoint='https://rest-catalog.example.com';
CALL iceberg_token='<bearer-token>';
LOAD FROM 'default.events' (file_format='iceberg') RETURN count(*);

Testing

• iceberg/test/test_files/iceberg_rest_catalog.test
  • RestCatalogRequiresWarehouse (runs everywhere): dynamic load, option registration, deterministic missing-warehouse error, and filesystem fallback after clearing options (verified against the local lineitem_iceberg table, count 51793).
  • RestCatalogScan (skipped, like unity_catalog.test): documented happy path against a live local REST catalog (e.g. Lakekeeper at 127.0.0.1:8181) including 2-/3-part names, path fallback, time travel, and the unknown-catalog error.
• e2e run: 2 passed / 3 disabled locally; clang-format-18 clean; sibling delta/duckdb/unity_catalog extensions still build.

Docs are in the new iceberg/README.md.