Split out of #5134, where it came up in the comments and then took the thread over. That issue is about KMS-backed JWT signing; this is a separate problem and deserves its own tracking.
Problem
GitHub App rate limits are per installation, meaning per App per org. To raise the budget for a single org you need multiple Apps installed on that org, each with its own 5,000 req/hour bucket.
But GitHub delivers a separate webhook event per App for the same workflow_job. Each has a different installation.id and the same workflow_job.id. Without deduplication the webhook lambda dispatches the same job to SQS N times, and N runners are created for one job.
This makes deduplication a prerequisite for multi-App rate limit scaling, not a corner case. The workarounds that avoid it do not actually help:
- One webhook App, others API-only — does not raise the budget, because the limit is per installation and only the webhook App's installation is used for the events.
- Partition Apps across orgs — raises total budget across orgs but not for any single org, which is the case that needs it.
Possible approaches
- SQS FIFO with
MessageDeduplicationId = workflow_job.id — exactly-once within the 5 minute dedup window, no external state. Probably the cleanest, but FIFO queues have throughput limits and would be a significant change to the queue topology.
- Short-lived dedup table — DynamoDB with TTL keyed on
workflow_job.id, conditional write to claim. More moving parts, no throughput ceiling.
Worth noting #5132 already introduces a DynamoDB table for cross-Lambda token caching, so option 2 would not be the first such dependency.
Split out of #5134, where it came up in the comments and then took the thread over. That issue is about KMS-backed JWT signing; this is a separate problem and deserves its own tracking.
Problem
GitHub App rate limits are per installation, meaning per App per org. To raise the budget for a single org you need multiple Apps installed on that org, each with its own 5,000 req/hour bucket.
But GitHub delivers a separate webhook event per App for the same
workflow_job. Each has a differentinstallation.idand the sameworkflow_job.id. Without deduplication the webhook lambda dispatches the same job to SQS N times, and N runners are created for one job.This makes deduplication a prerequisite for multi-App rate limit scaling, not a corner case. The workarounds that avoid it do not actually help:
Possible approaches
MessageDeduplicationId = workflow_job.id— exactly-once within the 5 minute dedup window, no external state. Probably the cleanest, but FIFO queues have throughput limits and would be a significant change to the queue topology.workflow_job.id, conditional write to claim. More moving parts, no throughput ceiling.Worth noting #5132 already introduces a DynamoDB table for cross-Lambda token caching, so option 2 would not be the first such dependency.