Skip to content

Fix deadlock from taking db lock while holding authz lock in Authz() - #564

Open
wallrj-cyberark wants to merge 1 commit into
letsencrypt:mainfrom
wallrj-cyberark:fix-authz-lock-order-deadlock
Open

wallrj-cyberark wants to merge 1 commit into
letsencrypt:mainfrom
wallrj-cyberark:fix-authz-lock-order-deadlock

Conversation

@wallrj-cyberark

Copy link
Copy Markdown

Fixes #563.

Authz() held the authorization's write lock while it looked up the account, which takes the MemoryStore read lock. FindValidAuthorization takes the same locks in the opposite order. Once a writer such as AddAccount queues on the store lock, the whole store wedges. #563 has the details and a goroutine dump.

This change does both lookups before authz.Lock():

  • The order's account ID. authz.Order is set when the authorization is created and never changes, so it is safe to read before taking the authz lock. Reading it first also removes the authz-then-order lock order, which is the reverse of Order.GetStatus().
  • The requesting account, via validPOSTAsGET or getAcctByKey. Both branches did this lookup first anyway, so responses and error order do not change.

The authz lock now only covers reading and updating the authorization itself.

Testing

[with Claude]

@wallrj-cyberark wallrj-cyberark left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review: one note on why the order is read without the authz lock, and one place I left alone.

Comment thread wfe/wfe.go Outdated
wallrj-cyberark added a commit to wallrj-cyberark/cert-manager that referenced this pull request Oct 1, 2026
Pebble can deadlock when a new-order, an authorization and a
new-account request arrive at the same time. After that it accepts
connections but never answers, so every ACME e2e test in the run
times out. See letsencrypt/pebble#563.

The new commit is current upstream Pebble main, plus the existing
Ed25519 patch, plus the fix from
letsencrypt/pebble#564.

Also correct the comment: we build from cert-manager's fork of Pebble,
not @inteon's, and it now carries two patches.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
Comment thread wfe/wfe.go
Authz() held the authorization's write lock while it looked up the
account, which takes the MemoryStore read lock. FindValidAuthorization
takes the same two locks in the opposite order: it holds the MemoryStore
read lock while it read-locks every authorization. Once a writer such as
AddAccount queues on the MemoryStore lock, the three goroutines wait on
each other and every later request that touches the db hangs.

Authz() also read-locked the order while holding the authz lock, which
is the reverse of the order used by Order.GetStatus.

Do the order and account lookups before taking the authz lock.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
@wallrj-cyberark
wallrj-cyberark force-pushed the fix-authz-lock-order-deadlock branch from f81564c to 2eced24 Compare October 2, 2026 05:46
cert-manager-prow Bot pushed a commit to cert-manager/cert-manager that referenced this pull request Oct 2, 2026
Pebble can deadlock when a new-order, an authorization and a
new-account request arrive at the same time. After that it accepts
connections but never answers, so every ACME e2e test in the run
times out. See letsencrypt/pebble#563.

The new commit is current upstream Pebble main, plus the existing
Ed25519 patch, plus the fix from
letsencrypt/pebble#564.

Also correct the comment: we build from cert-manager's fork of Pebble,
not @inteon's, and it now carries two patches.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Richard Wall <richard.wall@cyberark.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Authz() holds the authz lock while taking the db lock, which can deadlock the whole store

2 participants