fix(eks): set resource requests/limits for Calico/Tigera components#343
fix(eks): set resource requests/limits for Calico/Tigera components#343amdove wants to merge 2 commits into
Conversation
The tigera-operator chart ships resources: {} and the operator's
built-in component defaults set no memory bounds, so the Calico pods
had no guaranteed reservations and no ceilings — a source of resource
contention on busy nodes.
Pin predictable reservations on the operator pod, calico-node,
calico-typha, calico-kube-controllers, and calico-apiserver, following
a memory-bounded / CPU-unbounded policy: memory request == limit
(non-compressible; guaranteed floor + bounded ceiling), CPU request
only with no CPU limit (avoids CFS-throttling the dataplane).
EKS only; AKS uses the managed Calico add-on.
|
@claude review |
|
Claude finished @amdove's task in 1m 0s —— View job Code Review
This is a clean, well-scoped change. The implementation is correct and the rationale is sound. A few observations:
The two helpers are a nice abstraction. One minor note: The Resource values The chosen values look reasonable for typical cluster sizes. One thing to verify if you haven't already: the tigera-operator chart's
Overall No issues blocking merge. The change is correct, well-tested, and appropriately scoped to EKS. The AKS limitation is clearly documented. Approved by stevenolen — LGTM. |
Address review feedback: note that calicoComponentOverride patches only the main container (not ephemeral initContainers like install-cni), and explain why calico-apiserver's resources live under apiServer rather than installation (they map to the separate APIServer CR).
Description
Sets resource requests and limits on the Calico/Tigera components installed by the
tigera-operatorHelm chart on EKS. This should resolve resource contention issues with Calico: previously the chart shippedresources: {}and the operator's built-in component defaults set no memory bounds, so these pods had no guaranteed reservations and no ceilings, leaving them to compete unbounded with workloads on busy nodes.Policy applied (from the review discussion):
calico-nodegets the largest memory bound and keeps the operator's built-in 250m CPU floor, since Felix memory scales with the number of endpoints/policies.Code Flow
All changes are in
deployTigeraOperator(lib/steps/eks_helpers.go), which builds thetigera-operatorHelm release for the EKS path. Two small helpers were added:calicoResources(cpuRequest, memory)builds the{requests: {cpu, memory}, limits: {memory}}block encoding the memory-bounded / CPU-unbounded policy.calicoComponentOverride(container, cpuRequest, memory)wraps a per-container resource override in the operator's component-deployment strategic-merge shape.The dataplane overrides (
calicoNodeDaemonSet,typhaDeployment,calicoKubeControllersDeployment) are added under the existinginstallationvalues (the same passthrough already used forcalicoNetwork/cni); the operator pod gets top-levelresources; andcalico-apiservergetsapiServer.apiServerDeployment(the chart defaultsapiServer.enabled: trueand PTD does not disable it).Scope — what this does and does not fix
EKS: fixed. PTD installs the
tigera-operatorHelm chart on the EKS path, so these reservations attach directly to the release. Past resource-contention problems we've seen on EKS clusters should be resolved by this, and applying sane, explicit defaults here is the right thing to do regardless — it replaces "unbounded / no reservations" with predictable, logical values.AKS: not fixed by this change. AKS clusters also run Calico, but it comes from the Azure-managed network-policy add-on, not a PTD-installed chart. That add-on does not expose resource requests/limits — Azure owns and reconciles the installation, so the components (
calico-node,calico-typha,tigera-operator) ship with no requests/limits and cannot be tuned by us. This is a known, open Azure limitation: Azure/AKS#3496. So recent resource-contention issues observed on AKS clusters will not be addressed here and need a separate approach (e.g. node right-sizing, or migrating the network-policy dataplane to Cilium, which Azure recommends).Watch after applying
calico-node(512Mi) andcalico-typha(256Mi) OOM/restart counts on the largest clusters and bump if needed.Category of change
Checklist
Testing
just format,go vet, andgo test ./steps/pass.TestAWSEKSDeployTigeraResourceRequestsasserting each component's request/limit values and the no-CPU-limit invariant.