Fix: native K8s sidecar exit-143 causes false task failure (REMOTE-2365) - #117
Closed
warp-agent-staging[bot] wants to merge 1 commit into
Closed
Fix: native K8s sidecar exit-143 causes false task failure (REMOTE-2365)#117warp-agent-staging[bot] wants to merge 1 commit into
warp-agent-staging[bot] wants to merge 1 commit into
Conversation
inspectPodFailure treated every init container non-zero exit code as a task failure without checking restartPolicy. Kubernetes native sidecars are init containers with restartPolicy: Always, and Kubernetes itself excludes their exit codes from pod/Job success determination. JVM-based services (ZooKeeper, Kafka, Elasticsearch) always exit with code 143 (SIGTERM + 128) on clean shutdown — triggering a false task failure. Fix 1 (inspectPodFailure): Build a name→restartPolicy map from pod.Spec.InitContainers and skip init containers with restartPolicy: Always when evaluating non-zero exit codes. Adds buildInitRestartPolicyMap helper. Fix 2 (race guard): When inspectPodFailure returns a non-nil failure from the pod watch handler, do a synchronous Job GET before reporting the error. If the Job is already marked Complete, return success — the Job state takes precedence over transient pod events that arrive before the Job controller stamps JobComplete. Tests added: - TestBuildInitRestartPolicyMap: unit test for the new helper - TestInspectPodFailureIgnoresNativeSidecarExit143: ZooKeeper + Kafka exit 143 → no error - TestInspectPodFailureReportsRegularInitContainerExit: regular init container (no restartPolicy) exit 1 → still reported as failure - TestInspectPodFailureIgnoresMixedNativeSidecarsAndPassedSetup: native sidecar exits 143 alongside a passing setup init container → no error - TestExecuteTaskSucceedsWhenNativeSidecarExits143BeforeJobComplete: end-to-end race simulation: pod watch fires sidecar exit-143 first, job watch delivers Complete 50ms later → outcome is success - TestExecuteTaskJobCompleteWinsOverTransientPodFailure: race guard test: pod watch fires PodFailed, Job GET returns Complete → success Fixes REMOTE-2365 Co-Authored-By: Oz <oz-agent@warp.dev>
warp-agent-staging
Bot
added
the
from-feedback-bot
Contributor
|
Superseded by #118. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
When Kubernetes native sidecars (JVM-based services like ZooKeeper, Kafka, Elasticsearch) are declared as
initContainerswithrestartPolicy: Always, the worker falsely reports task failure on shutdown. The kubelet SIGTERMs these sidecars after the main task container completes, and JVM processes always exit with code 143 (128 + SIGTERM). The worker logged an error even though the Kubernetes Job itself reachedCompleteand the UI showed green Done.Root Cause
Two bugs in
kubernetes.go:Bug 1 —
inspectPodFailure: Treated every init container with a non-zero exit code as a task failure without checkingrestartPolicy. Kubernetes itself excludes native sidecars (restartPolicy: Always) from pod/Job success determination, but the worker did not.Bug 2 — Race condition: The pod status update carrying the sidecar exit-143 arrived via the pod watch before the Job controller stamped
JobComplete. SinceinspectPodFailuresaw exit-143 first, the failure path won every time.Fix
Fix 1 (
inspectPodFailure): Build aname → restartPolicymap frompod.Spec.InitContainers. Skip init containers withrestartPolicy: Alwayswhen evaluating non-zero exit codes. AddsbuildInitRestartPolicyMaphelper.Fix 2 (race guard): When
inspectPodFailurereturns a non-nil failure from the pod watch handler, do a synchronous Job GET before reporting the error. If the Job is already markedComplete, return success — the Job state takes precedence over transient pod events that arrive before the Job controller stampsJobComplete.Tests Added
TestBuildInitRestartPolicyMap: unit test for the new helperTestInspectPodFailureIgnoresNativeSidecarExit143: ZooKeeper + Kafka exit 143 → no errorTestInspectPodFailureReportsRegularInitContainerExit: regular init container exit 1 → still reported as failureTestInspectPodFailureIgnoresMixedNativeSidecarsAndPassedSetup: native sidecar exits 143 alongside passing setup init container → no errorTestExecuteTaskSucceedsWhenNativeSidecarExits143BeforeJobComplete: end-to-end race: pod watch fires sidecar exit-143 first, job watch delivers Complete 50ms later → successTestExecuteTaskJobCompleteWinsOverTransientPodFailure: race guard: pod watch fires PodFailed, Job GET returns Complete → successRelated
Fixes REMOTE-2365
Conversation: https://staging.warp.dev/conversation/6c426788-7214-456f-a10c-1d4b8739b8fa
Run: https://oz.staging.warp.dev/runs/019faa1e-d87e-7dc2-9d25-000a0106248d
This PR was generated with Oz.