feat(system-tests): support API boundary node playnets on the local backend - #11208
Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR enables API boundary node “playnet” functionality (domain + trusted TLS) on the local system-test backend by adding a hermetic in-group DNS resolver and issuing an ephemeral CA/cert for API BNs, allowing several previously Farm-only tests (notably those requiring cloud-engine subnets) to run locally. It also introduces a production-facing change to allow nns_delegation_manager to optionally trust additional API BN roots via config (intended for system tests).
Changes:
- Add local-backend DNS + per-group API BN domain/TLS issuance to support API BN playnets without Farm.
- Plumb a new dev-only GuestOS setting/config field for extra API BN trust anchors, and use it in
nns_delegation_manager. - Enable multiple system tests on the local backend and update firewall test logic to account for pre-seeded global firewall rules.
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| rs/tests/nns/BUILD.bazel | Enables delete_subnet_test off Farm by removing the Farm-only backend constraint. |
| rs/tests/networking/firewall/firewall_correctness_test.rs | Reads existing global firewall rules before proposing updates (local backend isn’t empty). |
| rs/tests/networking/firewall/BUILD.bazel | Keeps firewall correctness test on Farm, updating rationale for why local backend differs. |
| rs/tests/networking/canister_http_socks_test.rs | Clarifies why a hostname (nip.io-style) is required for SOCKS outcalls. |
| rs/tests/networking/BUILD.bazel | Enables canister_http_socks_test locally and corrects CPU sizing/commentary; documents remaining Farm-only test. |
| rs/tests/message_routing/xnet/BUILD.bazel | Enables xnet_cloud_engine_isolation_test off Farm by removing backend pin. |
| rs/tests/driver/src/util.rs | Adds template variable for extra_api_boundary_node_trust_anchors_pem. |
| rs/tests/driver/src/driver/test_env_api.rs | Adds TopologySnapshot::firewall_rules helper for reading registry firewall state. |
| rs/tests/driver/src/driver/local_backend.rs | Expands dnsmasq role to RA/DHCPv4/DNS; adds local DNS record registration and nip.io synthesis. |
| rs/tests/driver/src/driver/ic.rs | Implements local playnet setup: assign local domains, issue ephemeral CA/leaf, register DNS records. |
| rs/tests/driver/src/driver/bootstrap.rs | Plumbs local playnet TLS material + extra trust anchors into GuestOS config generation. |
| rs/tests/driver/Cargo.toml | Adds dependency on network crate to share GuestOS DNS server constants. |
| rs/tests/driver/BUILD.bazel | Adds Bazel dep on //rs/ic_os/networking/network. |
| rs/tests/crypto/BUILD.bazel | Enables cloud_engine_canister_sig_test off Farm by removing backend pin. |
| rs/orchestrator/src/firewall.rs | Updates config rendering template inputs for the new trust-anchor field in tests. |
| rs/ic_os/networking/network/src/systemd.rs | Introduces IPV6_NAME_SERVERS constant + unit test to keep it in sync with networkd contents. |
| rs/ic_os/networking/network/BUILD.bazel | Adjusts crate visibility to allow system tests to depend on network. |
| rs/ic_os/config/types/src/lib.rs | Bumps config version and adds dev-only extra_api_boundary_node_trust_anchors_pem field. |
| rs/ic_os/config/types/compatibility_tests/fixtures/hostos_v1.16.0.json | Adds v1.16.0 fixture covering the new dev setting. |
| rs/ic_os/config/types/compatibility_tests/fixtures/guestos_v1.16.0.json | Adds v1.16.0 fixture covering the new dev setting. |
| rs/ic_os/config/tool/templates/ic.json5.template | Adds new http_handler config field to rendered IC config. |
| rs/ic_os/config/tool/src/guestos/generate_ic_config.rs | JSON-encodes PEM into template var and wires it into config rendering. |
| rs/http_endpoints/nns_delegation_manager/src/nns_delegation_manager.rs | Extends trust store with optional extra roots from config when contacting API BNs. |
| rs/config/src/http_handler.rs | Adds new optional config field with default None. |
| Cargo.lock | Records the new network dependency in the lockfile. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 26 changed files in this pull request and generated no new comments.
Suppressed comments (1)
rs/tests/networking/canister_http_socks_test.rs:162
- This cross-reference still uses the old method name, but this PR renames it to
LocalBackend::start_dnsmasq; as written, readers cannot find the referenced implementation.
// `LocalBackend::start_ra_daemon`), so no external DNS is involved there.
Collaborator
Author
|
Picking up the suppressed comment from the last Copilot review ( Fixed in 6c287ca, along with three comments in |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
This pull request changes code owned by the Governance team. Therefore, make sure that
you have considered the following (for Governance-owned code):
-
Update
unreleased_changelog.md(if there are behavior changes, even if they are
non-breaking). -
Are there BREAKING changes?
-
Is a data migration needed?
-
Security review?
How to Satisfy This Automatic Review
-
Go to the bottom of the pull request page.
-
Look for where it says this bot is requesting changes.
-
Click the three dots to the right.
-
Select "Dismiss review".
-
In the text entry box, respond to each of the numbered items in the previous
section, declare one of the following:
-
Done.
-
$REASON_WHY_NO_NEED. E.g. for
unreleased_changelog.md, "No
canister behavior changes.", or for item 2, "Existing APIs
behave as before.".
Brief Guide to "Externally Visible" Changes
"Externally visible behavior change" is very often due to some NEW canister API.
Changes to EXISTING APIs are more likely to be "breaking".
If these changes are breaking, make sure that clients know how to migrate, how to
maintain their continuity of operations.
If your changes are behind a feature flag, then, do NOT add entrie(s) to
unreleased_changelog.md in this PR! But rather, add entrie(s) later, in the PR
that enables these changes in production.
Reference(s)
For a more comprehensive checklist, see here.
GOVERNANCE_CHECKLIST_REMINDER_DEDUP
|
✅ No security or compliance issues detected. Reviewed everything up to 37395b0. Security Overview
Detected Code Changes
|
|
✅ No security or compliance issues detected. Reviewed everything up to 37395b0. Security Overview
Detected Code Changes
|
daniel-wong-dfinity-org-twin
approved these changes
Aug 19, 2026
pierugo-dfinity
approved these changes
Aug 19, 2026
blind-oracle
approved these changes
Aug 19, 2026
Contributor
|
UPDATE: This PR of mine that adds a new system test has merged into master.
here is another TODO for this PR, I think? |
rzakrzyk
approved these changes
Aug 21, 2026
basvandijk
force-pushed
the
bas/local-backend-api-bn-playnet
branch
from
August 21, 2026 10:39
cb57a27 to
8dcfd1b
Compare
…ackend `with_api_boundary_nodes_playnet` gives API boundary nodes a domain name and a certificate for it that the nodes trust. Both halves came from Farm, so every test using it — and therefore every test with a cloud engine subnet, which requires it — was pinned to `backend = "farm"`. Stand up an equivalent locally: * Turn the group's `dnsmasq` into its DNS server. It already ran on the group bridge as an RA/DHCPv4 daemon with `--port=0`; dropping that and adding `--no-resolv --no-hosts` makes it a hermetic resolver that answers from an `--addn-hosts` file (new `LocalBackend::add_dns_record`) and from a `--synth-domain` mirroring the public `nip.io` wildcard service. GuestOS has no name-server knob and boots with `IPv6AcceptRA=no`, so rather than reconfiguring the guests, `create_group` assigns the four addresses GuestOS is hard-coded to query to the bridge. Inside the backend's own network namespace those addresses are free and no query can escape, so every node gets a working resolver without touching IC-OS. * Add `InternetComputer::setup_api_bn_local_playnet`, which issues an ephemeral CA plus a leaf covering the API boundary nodes' domains and registers those domains with the group's `dnsmasq`. `bootstrap` serves the leaf from `ic-boundary` through the existing `ic_boundary_tls_cert` mechanism. * Let the replica trust that CA. `nns_delegation_manager` built its root store from the compiled-in public roots only, which no test-issued certificate can satisfy. It now also honours `extra_api_boundary_node_trust_anchors_pem`, a new dev-only `GuestOSDevSettings` field that is unset in production, leaving the public roots as the only anchors there. Drops `backend = "farm"` from `canister_http_socks_test`, `cloud_engine_canister_sig_test`, `xnet_cloud_engine_isolation_test`, `nns_delegation_branch_nns_version_test` and `delete_subnet_test`. The `cpus` of `canister_http_socks_test` was understated: its comment omitted the four cloud engine nodes. Two tests stay on Farm for unrelated reasons, now recorded accurately: `nns_delegation_mainnet_nns_version_test` runs the mainnet GuestOS, whose replica predates the new field; and `firewall_correctness_test` asserts that port 8080 is closed between certain nodes, which the local backend cannot satisfy because `ic-prep` always adds 8080 to the `fd00::/8` rule the driver needs to reach the nodes. That test did assume the global firewall rule set starts out empty, which it does not on the local backend, so it now reads the current rules through the new `TopologySnapshot::firewall_rules`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`Url::parse` was handed the whole comma-separated list, which yields a single `Url` whose host is `icp-api.io,https` and whose path is the remainder. That re-serialises to `https://icp-api.io,https//icp0.io,https://ic0.app` — not a valid URL, and not the three the fixture meant to carry. Split it into three, and regenerate the v1.16.0 fixtures, which this branch introduces. The older fixtures keep the malformed value: they are historical records that must stay byte-for-byte as generated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…kend `start_ra_daemon`/`stop_ra_daemon` became `start_dnsmasq`/`stop_dnsmasq` when the daemon took on DNS, but a cross-reference in `canister_http_socks_test` still pointed at the old name, and three comments still called it the RA daemon even though it now serves RA, DHCPv4 and DNS. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…es plumbing `firewall_correctness_test` stays `backend = "farm"`, and on Farm the global firewall scope starts out empty, so reading the current rules and passing them as the proposal's `previous_rules` buys nothing there. It was a remnant of trying to also enable the test on the local backend, where the backend seeds a global rule so the driver can reach the nodes. Move it, and the `TopologySnapshot::firewall_rules` accessor it needed, to the follow-up that actually enables the test locally. Both files are byte-identical to master again. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… local upgrades `assert_assigned_replica_version` allows a node 600s to come up on a new replica version. On Farm that covers a whole GuestOS upgrade cycle; on the local backend it does not. This test's 10 VMs ask for 60 vCPUs and 40 GiB of guest RAM from a single host, and one cycle was measured at ~9 min there: ~90s to download the ~580 MiB update image, ~240s for `manageboot.sh` to `tar`-unpack it into the guest's tmpfs `/tmp`, ~35s to `dd` it onto the inactive slot and ~150s to reboot. Step 5's deadline expired 79s after the orchestrator came back up on the new version, before any replica had bound :8080, so the test panicked with "Replica did reboot, but never came back online!" -- which is only what `assert_assigned_replica_version_with_time` prints when its last poll errored. Wait 20 min per node instead, and raise the per-test timeout from 30 to 50 min: `ImageUpgrader::execute_upgrade` deletes the update image after installing and never checks whether the target version already sits on the inactive slot, so the roll back in Step 7 and the roll forward in Step 9 each pay for the full cycle again. A local run now passes in 2300s, still well inside the `test_timeout = "eternal"` hour the BUILD file gives the action. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
basvandijk
force-pushed
the
bas/local-backend-api-bn-playnet
branch
from
August 21, 2026 12:32
664477c to
37395b0
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
with_api_boundary_nodes_playnetgives API boundary nodes a domain name and a certificate for it that the nodes trust. Both halves came from Farm, so every system-test using it — and therefore every system-test with a cloud engine subnet, which requires it — was pinned tobackend = "farm".This stands up an equivalent locally.
DNS in the local backend
The group's
dnsmasqbecomes its DNS server. It already ran on the group bridge as an RA/DHCPv4 daemon with--port=0; dropping that and adding--no-resolv --no-hostsmakes it a hermetic resolver — with no upstream left to forward to, anything it cannot answer isREFUSED. It answers from two sources: an--addn-hostsfile, written by the newLocalBackend::add_dns_recordand re-read onSIGHUP, and a--synth-domainthat mirrors the publicnip.iowildcard service (socanister_http_socks_testneeds no DNS-specific code and stays identical on both backends).GuestOS has no name-server knob and boots with
IPv6AcceptRA=no, so rather than reconfiguring the guests,create_groupassigns the four addresses GuestOS is hard-coded to query to the bridge. Inside the backend's own network namespace those addresses are free and no query can escape, so every node gets a working resolver without touching IC-OS. Those addresses are now a singleIPV6_NAME_SERVERSconstant, kept in sync with the networkd contents by a unit test.A local playnet
InternetComputer::setup_api_bn_local_playnetissues an ephemeral CA plus a leaf covering the API boundary nodes' domains, registers those domains with the group'sdnsmasq, and stores the material in a newLocalApiBoundaryNodesPlaynetattribute — deliberately separate fromPlaynet, which the IC gateway VM also reads and writes.bootstrapserves the leaf fromic-boundarythrough the existingic_boundary_tls_certmechanism, so that pipeline is unchanged.The one production change
nns_delegation_managerbuilt its root store from the compiled-inwebpki_rootsonly, with no config, env var, feature orcfghook — no test-issued certificate could ever satisfy it. It now also honoursextra_api_boundary_node_trust_anchors_pem, a new dev-onlyGuestOSDevSettingsfield (config version 1.15.0 → 1.16.0). Production nodes getNoneand keep the public roots as their only trust anchors.This is the part that needs a careful look from the delegation-manager owners.
Tests enabled
backend = "farm"dropped fromcanister_http_socks_test,cloud_engine_canister_sig_test,xnet_cloud_engine_isolation_test,nns_delegation_branch_nns_version_testanddelete_subnet_test. Thecpusofcanister_http_socks_testwas understated — its comment omitted the four cloud engine nodes.Two tests stay on Farm for unrelated reasons, now recorded accurately in place of the old comment:
nns_delegation_mainnet_nns_version_testboots the mainnet GuestOS, whose replica predates the new field. It can be enabled once that has rolled out.firewall_correctness_testasserts that port 8080 is closed between particular pairs of nodes, but on the local backendic-prepalways adds 8080 to thefd00::/8rule the driver needs to reach the nodes at all — and the nodes themselves live infd00::/8. Enabling it means narrowing that rule to the driver's own addresses, which changes shared local-backend behaviour for every local test and is left as a follow-up.Verification
Run on the local backend, all passing:
cloud_engine_canister_sig_test_local,xnet_cloud_engine_isolation_test_local,nns_delegation_branch_nns_version_test_local,delete_subnet_test_local. Their logs show a cloud engine node resolvingapibn-0.ic.netthrough the group'sdnsmasq, completing a TLS handshake against the ephemeral CA, and fetching its NNS delegation.canister_http_socks_test_localis enabled but was not run: at 11 node VMs plus a UVM it needs ~48 GiB, more than the machine I verified on had.🤖 Generated with Claude Code