fix: bootstrap trusted kubelet serving certificates - #85
Open
r3loac wants to merge 3 commits into
Open
Conversation
- Add a Status printer column derived from the NodePool Ready condition - Regenerate the NodePool CRD - Add regression coverage for the printer column - Document the design, behavior, and deployment workflow
- Request a kubernetes.io/kubelet-serving certificate for the manager Pod IP - Replace the self-signed certificate after CSR approval without restarting - Renew serving certificates before expiration - Add the required Pod identity environment variables and CSR RBAC - Document CSR approval and kubelet TLS configuration - Add coverage for certificate installation and IP SAN validation
InftyAI-Agent
added
needs-triage
Member
|
/assign |
kerthcet
reviewed
Aug 31, 2026
kerthcet
left a comment
kerthcet
left a comment
Member
There was a problem hiding this comment.
Some comments since I can still not shell into the Modal workloads yet, one more thing, let's rebase the main branch since only the third commit is solve the cert issue.
Thanks!
| "serves TLS without client verification, because which CA signs the API server's kubelet "+ | ||
| "client certificate is not portable across distributions; restrict the port with a "+ | ||
| "NetworkPolicy, or set this to your API server's kubelet client CA.") | ||
| flag.BoolVar(&kubeletServingTLSBootstrap, "kubelet-serving-tls-bootstrap", true, |
Member
There was a problem hiding this comment.
Let's disable this by default.
| --sort-by=.metadata.creationTimestamp -o name | tail -n1) | ||
|
|
||
| # Confirm the requested IP SAN matches the manager Pod IP before approving it. | ||
| kubectl get csr "$CSR" -o jsonpath='{.spec.request}' \ |
Member
There was a problem hiding this comment.
This is not right, no need to specify the resource type for $CSR.
| | grep 'installed trusted kubelet serving certificate' | ||
| ``` | ||
|
|
||
| An installation with an external CSR approver should restrict it to requests that |
Member
There was a problem hiding this comment.
Have you tried yourself, I tested this by following the instructions here but seems still error:
kg csr
nebula-kubelet-serving-c07b251cec59bb78b897a236 15m kubernetes.io/kubelet-serving system:serviceaccount:nebula-system:nebula-controller-manager 30d Approved
kubectl -n nebula-system logs deploy/nebula-controller-manager \
| grep 'installed trusted kubelet serving certificate'
# null
keti ky-modal-demo-qevhkl6-sandbox-bcc84cbd8-hrllq -- /bin/bash
error: Internal error occurred: error sending request: Post "https://10.20.15.172:10250/exec/org-d3d871f4-9d1d-4df5-85cb-0b1472f51857/ky-modal-demo-qevhkl6-sandbox-bcc84cbd8-hrllq/sandbox?command=%2Fbin%2Fbash&input=1&output=1&tty=1": tls: failed to verify certificate: x509: certificate signed by unknown authority
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Nebula now requests a trusted kubelet serving certificate through the
kubernetes.io/kubelet-servingCSR signer.The CSR contains the manager Pod IP in its SAN. After approval and issuance,
Nebula atomically replaces the self-signed fallback certificate without
restarting the kubelet API or interrupting existing streams.
Why
Managed Kubernetes control planes such as EKS may verify kubelet serving
certificates. Nebula previously used only a self-signed certificate, causing
kubectl logsandkubectl execto fail with:x509: certificate signed by unknown authorityAdditional changes
POD_NAMEandPOD_UIDprojectionsVerification
go test ./pkg/vnode ./cmd -count=1go vet ./...Deployment note
The
kubernetes.io/kubelet-servingsigner requires an external approvaldecision. Approve the generated CSR with: