test(k8s): pin Kubernetes workload identity against a real cluster

[lakhansamani]

Asked to make sure nothing breaks in Kubernetes and to add k3d/kind tests. The
tests found something bigger than the change that prompted them.

The finding

In-cluster Kubernetes workload identity cannot work. Not TokenReview
specifically — the whole path.

Measured against a real kind cluster:

What the cluster publishes	Address	SafeHTTPClient
issuer	https://kubernetes.default.svc.cluster.local	ClusterIP → refused
jwks_uri	https://172.27.0.2:6443/openid/v1/jwks	RFC 1918 → refused
apiserver	https://127.0.0.1:61411	loopback → refused

validators.SafeHTTPClient rejects private, loopback and link-local addresses
unconditionally, and clientauth has no SafeHTTPClientAllowPrivate escape
hatch (the two that exist are gated on --env=e2e and belong to oauth_sso and
webhook delivery).

Meanwhile performTokenReview reads Authorizer's own in-cluster
ServiceAccount token (/var/run/secrets/kubernetes.io/serviceaccount/token) to
authenticate to an apiserver it can never dial. The two assumptions contradict
each other.

performTokenReview already documented half of this. The JWKS half was
undocumented and is the wider problem: it applies with or without TokenReview,
because static_jwks_url and oidc_discovery both point at the cluster's own
private address. That is why this is not a kind artefact — kubernetes.default.svc
is always a ClusterIP, and jwks_uri always points at the apiserver.

The operator-visible symptom, captured by the test:

400 {"error":"invalid_client","error_description":"Client authentication failed"}

Nothing points at the refused fetch. Worth fixing whenever the address problem is.

Why these tests, and why they assert failure

They assert current behaviour, deliberately. They are a pin, not an
aspiration: if someone changes the SSRF policy, the suite says precisely which
Kubernetes behaviour they changed. TestK8sWorkloadAuthenticationEndToEnd names
itself as the assertion to invert once this is fixed.

Three cases, and the middle one earns its place: TestK8sProjectedTokenIsWellFormed
proves the token side is correct — real iss, sub, audience-bound aud — so
the end-to-end failure is attributable to the fetch address and nothing else.
Without it a reader could reasonably blame the token.

Mechanics

• scripts/k8s-e2e.sh — kind by default, K8S_RUNTIME=k3d for k3d, K8S_KEEP=1
  to leave the cluster up. Always tears down, including on failure.
• make test-k8s
• .github/workflows/k8s.yml — scheduled weekly plus PRs touching the code that
  decides cluster reachability. Not per-push: it provisions a cluster to guard a
  limitation that changes rarely.
• Behind the k8s build tag. Verified make test is unaffected — 43 packages,
  0 FAIL, and the file is not compiled into normal runs.

Verification

Cold run from no cluster: create → test → teardown, exit 0, no cluster left behind.

go build ./...                          OK
go vet ./... && go vet -tags k8s ./...  OK
make test                               exit 0 — 43 packages, 0 FAIL
make lint                               exit 0
make test-k8s                           exit 0 — 3 tests, real kind cluster

Not fixed here

Making in-cluster workload identity actually work needs a security decision, not
a patch: a scoped SSRF exemption for an operator-declared apiserver/JWKS host
with CA pinning, or an explicit in-cluster transport. SafeHTTPClientAllowPrivate
asks for careful review before a third caller, and this would be one. README now
states the constraint instead of listing the feature unqualified.

[lakhansamani]

Pushed a correction to this PR. The tests I opened it with were wrong, and passed only because CI runs kind.

The defect

TestK8sClusterAddressesAreRefusedBySSRFGuard asserted unconditionally that a cluster's issuer, jwks_uri and apiserver are private, and TestK8sWorkloadAuthenticationEndToEnd asserted a projected token therefore cannot authenticate. Both generalise from kind, which uses the default --service-account-issuer.

EKS, GKE and AKS publish a public https issuer with public OIDC discovery by default. Probed against the real guard:

https://oidc.eks.us-west-2.amazonaws.com/id/EXAMPLE/.well-known/openid-configuration  -> err=<nil>
https://container.googleapis.com/v1/projects/p/locations/l/clusters/c                 -> err=<nil>

So on the major managed distributions this feature works today with no configuration beyond the trusted issuer — and my assertions would fail there, demanding a refusal that correctly did not happen.

The fix

Both tests now derive their expectation from what the cluster actually publishes, resolving the address independently of the code under test:

• private address → MUST be refused, workload cannot authenticate
• public address → MUST be accepted, workload MUST authenticate

The suite is now truthful on whatever cluster it is pointed at, and states the rule rather than one cluster's instance of it. Still passes on kind (both addresses private, both branches take the refusal path).

Docs corrected too

README said "requires a publicly-routable issuer/JWKS endpoint", which reads as "this does not work" when for EKS/GKE/AKS it is the default. It now names which clusters work untouched, and gives the one-line fix for the rest: point jwks_url at a reachable mirror of /openid/v1/jwks, because issuer_url is only matched against the token's iss and is never fetched — traced to client_assertion.go:149 (GetTrustedIssuerByIssuerURL(ctx, iss)) and :413 (the only fetch, in the oidc_discovery branch), and confirmed by probe.

performTokenReview's doc comment carried the same overstatement and is corrected identically.

Verification

go build ./...              OK
go vet ./... (+ -tags k8s)  OK
make test                   exit 0 — 43 packages, 0 FAIL
make lint                   exit 0
make test-k8s               exit 0 — cold run, create → test → teardown