Skip to content

feat: support per-runner-flavor SQS batch size and window in multi_runner_config#5108

Merged
Brend-Smits merged 2 commits into
mainfrom
copilot/support-sqs-batch-size-window-config
Jul 3, 2026
Merged

feat: support per-runner-flavor SQS batch size and window in multi_runner_config#5108
Brend-Smits merged 2 commits into
mainfrom
copilot/support-sqs-batch-size-window-config

Conversation

Copilot AI commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Description

lambda_event_source_mapping_batch_size and lambda_event_source_mapping_maximum_batching_window_in_seconds are module-level variables applied identically to every runner flavor. Deployments with mixed load profiles (e.g. 1000-runner high-volume vs 10-runner low-volume) need per-flavor tuning to avoid unnecessary latency, wasted SSM throughput, and GitHub API rate limit pressure on low-volume flavors.

Adds both as optional(number, null) fields in multi_runner_config.runner_config, with coalesce() fallback to the existing module-level variables. Follows the established pattern used by scale_up_reserved_concurrent_executions.

Changes:

  • modules/multi-runner/variables.tf — new optional fields in the runner_config object type + descriptions; restored iam_overrides fields accidentally dropped during merge
  • modules/multi-runner/runners.tfcoalesce(per_flavor, module_level) for both parameters; restored iam_overrides wiring
  • modules/multi-runner/README.md — regenerated with terraform-docs to reflect all changes; kept in sync with latest main

Usage:

multi_runner_config = {
  runner-large = {
    runner_config = {
      lambda_event_source_mapping_batch_size                         = 50   # override
      lambda_event_source_mapping_maximum_batching_window_in_seconds = 10   # override
      # ...
    }
  }
  runner-metal = {
    runner_config = {
      # no override — uses module-level defaults
      # ...
    }
  }
}

Fully backwards compatible. Existing configs require no changes.

Test Plan

  • Verified with terraform fmt (clean)
  • CodeQL security scan passed (no issues)
  • Code review passed (no issues found)
  • Confirmed iam_overrides is fully restored in runner_config object type, heredoc description, standalone variable, and runners.tf wiring
  • Merged latest main and regenerated README; no remaining conflicts

Related Issues

Copilot AI changed the title [WIP] Support per-runner-flavor SQS batch size and window configuration feat: support per-runner-flavor SQS batch size and window in multi_runner_config Apr 14, 2026
Copilot AI requested a review from Brend-Smits April 14, 2026 08:32
@github-actions

github-actions Bot commented Apr 14, 2026

Copy link
Copy Markdown
Contributor

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

@wadherv

wadherv commented May 18, 2026

Copy link
Copy Markdown
Contributor

@Brend-Smits can you please review this PR?

@Brend-Smits

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

@Brend-Smits
Brend-Smits requested a review from Copilot June 10, 2026 14:16
@Brend-Smits
Brend-Smits marked this pull request as ready for review June 10, 2026 14:16
@Brend-Smits
Brend-Smits requested a review from a team as a code owner June 10, 2026 14:16

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds per-runner-flavor overrides for SQS event source mapping tuning in the multi-runner module, allowing mixed runner fleets to use different Lambda batching behavior without changing global defaults.

Changes:

  • Extends multi_runner_config.runner_config with per-flavor optional overrides for lambda_event_source_mapping_batch_size and lambda_event_source_mapping_maximum_batching_window_in_seconds.
  • Updates the per-flavor module "runners" wiring to coalesce(per_flavor_override, module_level_default) for both settings.
  • Regenerates/updates modules/multi-runner docs to reflect the new fields.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
modules/multi-runner/variables.tf Adds optional per-flavor batch size/window fields to the runner_config object type and documents them in the multi_runner_config heredoc.
modules/multi-runner/runners.tf Applies per-flavor overrides via coalesce() when passing event source mapping settings into the runners module.
modules/multi-runner/README.md Updates the generated inputs documentation to include the new per-flavor configuration options.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread modules/multi-runner/variables.tf Outdated
Comment thread modules/multi-runner/variables.tf Outdated
Comment thread modules/multi-runner/README.md Outdated

Copilot AI commented Jun 10, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Resolved the merge conflict in modules/multi-runner/README.md by merging origin/main and regenerating the terraform-docs section. Pushed in commit e530643.

@Brend-Smits
Brend-Smits force-pushed the copilot/support-sqs-batch-size-window-config branch from 4cddd3c to aaa109f Compare June 10, 2026 15:01
@Brend-Smits
Brend-Smits requested a review from a team as a code owner June 10, 2026 15:01
@Brend-Smits
Brend-Smits force-pushed the copilot/support-sqs-batch-size-window-config branch from aaa109f to 757a59a Compare June 10, 2026 15:05
@edersonbrilhante

Copy link
Copy Markdown
Contributor

@Brend-Smits this PR is removing the iam_overrides added in the PR #4875

@Brend-Smits

Copy link
Copy Markdown
Contributor

@Brend-Smits this PR is removing the iam_overrides added in the PR #4875

Good catch, thanks. Must have done something wrong when merging.

Comment thread modules/multi-runner/variables.tf
Comment thread modules/multi-runner/variables.tf
Comment thread modules/multi-runner/variables.tf
@Brend-Smits

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Merged origin/main into the branch and regenerated modules/multi-runner/README.md with terraform-docs. No conflicts remain.

@edersonbrilhante

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

1 similar comment
@Brend-Smits

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

Copilot AI commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

@copilot resolve the merge conflicts in this pull request

Merged origin/main into the branch. The only conflict was in modules/multi-runner/README.md on the multi_runner_config row — resolved by keeping the HEAD version (which includes our new lambda_event_source_mapping_batch_size and lambda_event_source_mapping_maximum_batching_window_in_seconds fields along with all changes from main). No conflicts remain.

@copilot resolve the merge conflicts in this pull request

Merged origin/main into the branch in commit 44d4106c. Resolved the single conflict in modules/multi-runner/README.md by keeping the HEAD version that includes our new per-flavor SQS batch size/window fields plus all main changes. No conflicts remain.

@edersonbrilhante edersonbrilhante left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Brend-Smits Can you fix the non signed commit, otherwise we will need another person to review

@Brend-Smits
Brend-Smits force-pushed the copilot/support-sqs-batch-size-window-config branch from 049474b to 5e0dac2 Compare July 3, 2026 15:19
@edersonbrilhante

Copy link
Copy Markdown
Contributor

@copilot resolve the merge conflicts in this pull request

@Brend-Smits
Brend-Smits force-pushed the copilot/support-sqs-batch-size-window-config branch from fefe5bd to f936c99 Compare July 3, 2026 15:27
…nner_config

Add configurable SQS batch_size and maximum_batching_window_in_seconds
per runner flavor in the multi_runner_config. This allows fine-tuning
how many messages are consumed per Lambda invocation and how long to
wait before invoking, improving cost efficiency and scaling behavior
for different runner types.

Signed-off-by: Brend Smits <brend.smits@philips.com>
@Brend-Smits
Brend-Smits force-pushed the copilot/support-sqs-batch-size-window-config branch from a59afff to 04ef0e3 Compare July 3, 2026 15:29
@Brend-Smits

Copy link
Copy Markdown
Contributor

@Brend-Smits Can you fix the non signed commit, otherwise we will need another person to review

Resolved! Please have another look and feel free to merge then.

@Brend-Smits
Brend-Smits enabled auto-merge (squash) July 3, 2026 15:31
@Brend-Smits
Brend-Smits merged commit ecfdfdb into main Jul 3, 2026
42 checks passed
@Brend-Smits
Brend-Smits deleted the copilot/support-sqs-batch-size-window-config branch July 3, 2026 15:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support per-runner-flavor SQS batch size and window configuration in multi_runner_config

5 participants