Add: TMR kernel resident arguments and invocation isolation（K5）

[Leaf-Salix]

Stacked on #2180 at b0943525dd8eaeb486bca92bbf5480a37eb61fcb. The inherited stack contains K1 (#2064), TMR resource contracts (#2177), and the K2 integration from #2176. Review the K5 delta against this fixed K4 baseline; changes inherited from those prerequisites are not K5 additions. The HBG/2b implementation is outside this stack.

This branch labels prerequisite commit subjects by component without changing their trees. Its equivalent K4 parent is c73001d69b78b84a238b2e78be1102f984f6c0e9; the K5 commit is b6435e48. K5-only diff.

Draft scope: resident-argument preparation and independently testable TMR execution-input consumption. Public kernel execution and graph replay still require the resource providers and execution binder. Reconcile prerequisite changes and rerun the affected validation before merge.

Motivation

K4 supplies an invocation snapshot, but the TMR executor also needs context-stable configuration and persistent AICore entry arguments. Passing each invocation by rewriting a shared Runtime would mix these lifetimes and undermine asynchronous graph execution.

This PR separates their sources while retaining the existing KernelArgs/Runtime layouts and reusing the executor, scheduler, and K2 resource owner:

init configuration copy
  → prepare-time topology resolution
  → persistent Runtime prefix, register table, and KernelArgs

K4 packet + trusted callable/binding views
  → admission into executor-owned argument storage
  → config and orchestration consume the same invocation
  → scheduler resolves functions through this callable's bounded table
  → existing child task payloads
  → release references after all consumers have finished

The two lifetimes remain separate: invocation processing does not update the resident callable ID, function table, or entry-argument fields. Workers and teardown gates remain mutable execution state; this PR does not claim the entire Runtime is immutable.

Resident contract and execution adaptation

Context configuration and persistent arguments

src/common/platform/include/host/kernel_static_config.h introduces KernelStaticConfig, a Host-owned copy of the context request and its generation. Packed input is read through memcpy; the caller's configuration pointer is not retained. Invalid thread counts are rejected. Kernel mode snapshots the existing SIMPLER_TMR_SERIAL_ORCH_SCHED_ENABLE setting at init; program mode retains its existing configuration behavior.

The first prepare resolves worker/thread counts and architecture-specific affinity before publishing persistent arguments. Later prepares reuse the fixed configuration and resident allocations. Kernel diagnostics requests remain explicitly unsupported, with resident DFX fields zeroed. The a2a3 FFTS fields and a5 SIMT anchor are preserved.

DeviceRunnerBase::prepare_kernel_callable() performs topology resolution and runtime-specific static configuration before calling the existing PersistentKernelArgs::prepare_once(). Configuration is fixed at the context level, not selected by the first or last registered callable. The prepare ABI does not accept replacement configuration, and invocation consumption does not update it.

The resident TMR upload remains the DeviceRuntimeLaunchDesc prefix of Runtime, not its Host-only members. The physical layout is unchanged: callable ID, function table, orchestration arguments, and execution-memory placeholders retain their empty/sentinel values. Their presence in the layout does not make them kernel invocation inputs.

Persistent ownership and failure recovery

src/common/platform/onboard/host/kernel_persistent_args.{h,cpp} retains the existing PersistentKernelArgs owner for the Runtime image, register-address table, and device KernelArgs. Its failure handling is tightened:

• Record ownership as allocations succeed, including partial architecture initialization.
• Expose ready arguments only after preparation succeeds completely.
• Revoke readiness on finalize; retain pointers whose release failed for a later close retry.
• Prevent another prepare or allocator-wide cleanup from discarding partially retained ownership.
• Keep abandonment restricted to the existing device-resource invalidation contract.

The a5 kernel topology query also retains its temporary allocation and pending stream after submission/synchronization failure. Cleanup must establish completion before freeing that buffer. A failed free on the successful query path must not publish cached topology or report success. This handling does not reset a caller-owned device.

TMR-specific invocation views and consumers

src/common/tensormap_and_ringbuffer/kernel_execution_inputs.h defines internal borrowed views, not another wire format or resource registry:

Input	Responsibility
KernelBindingView	Trusted binding identity, resident Runtime, SM/arena regions, and RuntimeContext offset
KernelCallableView	Trusted callable identity/counts/generation and a bounded CoreCallable address table
ExecutionInputs	Current callable ID, materialized arguments, function table, and explicit execution-memory sources

Admission checks region bounds/alignment, size arithmetic, RuntimeContext placement, and function-table bounds before consuming the K4 packet. K4 validates the packet against the independent trusted identities. The provider must already supply a valid initialized image and keep all borrowed resources alive; matching generations alone do not establish lifetime safety. device_binding_addr remains opaque.

Each existing AicpuExecutor owns one KernelInvocationState with its argument storage. The internal phases are admit_kernel_execution, init_kernel_execution, run_kernel_execution, kernel_execution_status, and release_kernel_execution. An active invocation rejects replacement; this guard is not a concurrency lock.

InvocationStatus admit_kernel_execution(
    ByteSpan packet,
    const KernelCallableView& callable,
    const KernelBindingView& binding) noexcept;
int32_t init_kernel_execution();
int32_t run_kernel_execution();
int32_t kernel_execution_status();
void release_kernel_execution();

KernelInvocationState::admit() invokes K4's consume_tmr_invocation() once and materializes descriptors/scalars into executor-owned EntryArgsStorage. It does not copy Tensor data, replace K4 decoding, allocate a per-invocation device buffer, or create a large temporary Runtime on the stack. Failed admission publishes no execution inputs.

The implementations in both src/{a2a3,a5}/runtime/tensormap_and_ringbuffer/aicpu/aicpu_executor.cpp feed these inputs into the existing execution body. HBG does not consume this TMR view or acquire a TMR payload requirement.

Both architectures consume the explicit inputs throughout:

• Config and orchestration use the same callable ID and argument storage.
• Both scheduler handshake initialization stages retain the invocation's function table.
• Initial task counts and execution status use the bound SM, not resident placeholders.
• Ordinary, early, and gated dispatch reject invalid function mappings before publishing work.
• Final orchestration unbinding uses the current callable ID. Release clears scheduler/argument references before clearing invocation storage.

No new public C API, K9/K4 packet layout, device argument pool, or callable manager is introduced.

Lifecycle and program-path impact

Execution transaction: The caller owns the publication and completion protocol around the internal consumer phases:

acquire exclusive use of the executor/workspace
  → obtain independently trusted, live callable and binding views
  → admit and materialize this invocation
  → publish inputs to the execution threads
  → initialize and run the existing TMR consumers
  → complete or cancel execution; join all consumers
  → release borrowed references, then clear invocation storage
allow the next invocation to reuse the executor/workspace

Ownership: Config and orchestration borrow the same executor-owned argument storage through orch_args_cached_. On normal completion, the final execution thread unbinds the current callable's orchestration Runtime. Release resets scheduler references and cached arguments before clearing storage. The provider keeps the resident Runtime, arenas, SM, function table, and callable images alive through their last device use; a generation match is not a retain operation.

Serialization: Admission and release require one owner. The runtime DSO's executor is a shared static instance: protection must cover every user of that instance, including mixed program/kernel use, rather than only one context's lock. No additional mutex, registry, or implicit launch synchronization is introduced. A successful native enqueue is not evidence that storage can be reused.

Program compatibility: Program entry wrappers continue to obtain their inputs from the original Runtime. Program bind/staging and per-run resource preparation are not redirected through the kernel path. Shared executor/scheduler code is modified, so this is not a claim of zero program-source impact; valid program behavior is covered by regression testing. The a2a3 program cleanup retains its Runtime cache invalidation, while kernel cleanup does not assume that the next invocation will refresh the resident Runtime from Host memory.

Dependencies: K2 supplies the resident owner and context facilities; K4 supplies the packet and argument conversion. K3/K10 retain responsibility for execution-resource and callable publication/lifetime. K7/binder supplies platform entry setup, execution ordering, cancellation, completion, and reuse protection. This PR exposes callable consumption phases but does not register an incomplete public device entry. simpler_kernel_mode_supported() remains 0; public launch still rejects execution. Sim kernel init retains its validated unsupported endpoint, and HBG kernel init is not enabled by this TMR change.

Tests

Results describe the source snapshot published as b6435e48. Commit-subject relabeling preserved every source tree; it did not constitute a new test run. These results are not a completed GitHub CI run.

Coverage added in this PR

• tests/ut/cpp/common/test_kernel_persistent_args.cpp: Allocation/copy/partial-architecture failures, retained ownership after failed rollback, readiness revocation, retrying only remaining allocations, and abandonment. Shared owner tests cover TMR/HBG on both architectures.
• tests/ut/cpp/common/test_kernel_execution_inputs.cpp: Packed configuration ownership, replacement rejection, binding bounds/alignment, stale identity rejection, unchanged published inputs on failure, program storage borrowing, and empty-Tensor conversion compatibility.
• tests/ut/cpp/common/test_tmr_scheduler_execution_inputs.cpp: 6 cases per architecture, covering both handshake binding sites, A/B/A function selection, ordinary/early/gated payload publication, invalid mappings, and program wrappers.
• tests/ut/cpp/common/test_tmr_executor_execution_inputs.cpp: 4 cases per architecture, covering actual config/orchestration consumption, serial/nonserial A/B/A reuse, failed admission, early init rejection, and config failure followed by release and successful execution of another invocation.
• tests/ut/cpp/hardware/test_a5_kernel_topology_lifecycle.cpp: 9 cases using the real onboard Host implementation with injected runtime operations. They check allocation/copy/launch/sync/free failures, retained ownership, close retries, and the absence of device reset or ACL finalization.
• tests/ut/py/test_kernel_mode_c_api.py: Real sim entry checks for invalid and unsupported kernel configuration without enabling execution.

Validation results

Validation	Result and scope
C++ regression matrix	18 CTest targets passed, covering inherited contracts, four TMR/HBG resident-owner variants, both TMR makers, scheduler/executor consumers, and a5 topology lifecycle
Sanitizers	10 targets passed ASan/UBSan in the closure run
Executor failure-reuse supplement	Both architecture targets, now four GTests each, passed normally and under ASan/UBSan; normal execution repeated three times successfully
C ABI	29 passed, 36 deselected; deselected cases are not passes
Runtime build and symbols	Eight architecture/platform/runtime components rebuilt and kernel lifecycle symbols checked
Program simulation regression	Both architecture sweeps completed successfully; L2 a2a3 HBG: 12 passed/7 skipped, TMR: 30 passed/1 skipped; a5 HBG: 13 passed, TMR: 29 passed
Incremental lint	Applicable pre-commit checks passed for the full K5 set and subsequently for the two supplementary test files

The executor tests compile the real executor, scheduler, runtime, and SO loader. They exercise K4 admission followed by actual config/orchestration calls, serial/nonserial A/B/A reuse, transport-buffer overwrite, and resident-field preservation. The orchestration fixture submits a dependency-only task; actual AICore execution is not claimed. Child function-table selection is covered separately through the real scheduler's ordinary/early/gated paths.

The supplementary tests cover early init configuration rejection and config expected-count mismatch. Both threads return, release runs, and a new invocation succeeds with its own arguments. They do not establish recovery from every partial handshake or asynchronous device cancellation.

Topology fault injection exercises the actual a5 onboard Host implementation without NPU execution. Isolated builds use explicit ccache and four-way parallelism; existing Torch dependencies are not installed or replaced.

Reproduction

After the repository's isolated build setup, the focused executor tests can be rebuilt and run with:

export CCACHE_DIR="$PWD/.ccache"
export CMAKE_BUILD_PARALLEL_LEVEL=4
ccache -s
cmake -S tests/ut/cpp -B tests/ut/cpp/build \
  -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
cmake --build tests/ut/cpp/build --parallel 4 \
  --target test_a2a3_tmr_executor_execution_inputs \
           test_a5_tmr_executor_execution_inputs
ctest --test-dir tests/ut/cpp/build --output-on-failure --timeout 60 \
  -R '^test_(a2a3|a5)_tmr_executor_execution_inputs$'
ctest --test-dir tests/ut/cpp/build --output-on-failure --timeout 60 \
  --repeat until-fail:3 \
  -R '^test_(a2a3|a5)_tmr_executor_execution_inputs$'

Use a separate build directory with -fsanitize=address,undefined -fno-omit-frame-pointer for the sanitizer run. The a5 topology target additionally requires the existing hardware-test CMake option, CANN development dependencies, and the built a5 onboard Host library; its injected Host test does not execute an NPU kernel.

Remaining acceptance

• A5 silicon execution.
• Public kernel launch with real resource and residency providers.
• ACLGraph capture/replay, cross-stream exclusion, and graph-visible resource release.
• Full-pipeline steady-state HBM behavior and launch-time allocation/copy measurements.

These remain integration acceptance items. Host fixtures, owner counters, and program simulation do not substitute for them. The earlier K4 CPU transport probe is not counted as K5 executor or graph-execution evidence.

Commits

The K5 implementation is b6435e48 — Add: isolate TMR resident and invocation state（K5）: persistent ownership fixes, immutable context configuration, explicit invocation consumption in both executors/schedulers, and associated tests.

Earlier commits are prerequisite code, with component labels added to their subjects. Their source trees are unchanged:

Component	Commit in this branch	Original commit
K1	66156ed3	dc1268cd
2a resource requirements	9e5492b1	48e37b3b
2a interface alignment	db2553e7	2153406a
K4 snapshot foundation	b5529c89	3e822453
K2/K4 integration	c73001d6	b0943525

The dependency chain is K1 → {2a, K2} → K4 → this K5 PR. K2's implementation is already included in the K4 integration snapshot; this branch does not introduce a separate K2 merge commit. For a focused review, compare c73001d6...b6435e48 rather than attributing the entire branch-to-main diff to K5.

中文总结

• 合并依赖： 基于 Add: TMR kernel invocation snapshots and native transport adapter（K4） #2180 的 b0943525，包含 K1、2a 和 K2/K4 组合实现。当前 K5 提交为 b6435e48；前序提交仅整理标题，代码树不变。等待前置合入后对齐复验，保持 Draft。
• 交付内容： init 复制 context 配置，prepare 解析硬件拓扑并固定常驻参数；K4 快照转换进入真实 executor，config/orchestration 使用同一份本轮参数，scheduler 两处绑定均使用当前 callable 的函数表。
• ABI 与解耦： 保留现有公开 kernel ABI、K9/K4 wire 和 program KernelArgs/Runtime 布局；复用 K2 owner 和原 executor/scheduler，不另建参数池、allocator 或资源管理表。program 保留原数据来源，不能将共享源码改动表述为“完全没有影响面”。
• 生命周期： 常驻参数准备失败或 free 失败不丢账，撤销可借用状态并支持 close 重试；本轮 storage 在所有消费者结束后才释放引用和复用。generation 检查不代替资源保活，跨 context/跨模式共享 executor 的串行保护由 K7/binder 负责。
• 验证结果： 收尾矩阵 18 个 C++ target、29 个 ABI 用例、两架构 program sim 回归通过；10 个 target 的 ASan/UBSan 通过。补测后的真实 executor 两架构各 4 个用例，普通、三次重复及 sanitizer 均通过，新增覆盖 init/config 失败后 release 与成功复用。发布前全部修改文件的适用 pre-commit 检查通过，未安装 Torch。
• 未完成范围： A5 真机、真实 provider 与 K7/binder 接通后的公开 launch、ACLGraph capture/replay、并发取消及 graph 生命周期仍需组合验收。本 PR 提供独立可评审的常驻准备和消费实现，不宣称整条 kernel pipeline 已完成。

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}