Skip to content

[Bug] Explicit FFTS synchronization fails to compile on a2a3sim #2188

Description

@Little-oil

Platform

a2a3sim (Ascend 910B/C simulation)

Runtime Variant

All / Unknown

Description

Valid PyPTO programs using explicit FFTS cross-core synchronization compile
for A2/A3 hardware but fail during A2/A3 simulator kernel compilation. PTOAS
emits the device FFTS message helpers and synchronization builtins, while the
CPU simulator compilation environment provides no compatible definitions or
event semantics.

This currently blocks the DeepSeek V4 DSpark CSA, HCA, and SWA simulator
tests. Their generated AIC/AIV kernels use the same explicit sync_set and
sync_wait operations. Failure occurs before numerical execution.

Steps to Reproduce

Use the following matching dependency chain:

  • pypto-lib: 3daaee2c0b3f7ff274cc9980408287ad0bf47cec
  • PyPTO: a9b9e0be9730f6b5fe63fb151b6d4331f4d770c8
  • simpler: 39ce891dbb3f665e72e99b4a1387b47012fb18bd
  • PTOAS: v0.60
  • PTO ISA: 5a4f74cbf627d4aac2e0ce10d5e0d8b118343265

Run a DSpark attention entry on CPU simulation:

python models/deepseek_v4_flash_dspark/decode_csa.py -p a2a3sim

The same compile failure is reproduced by a minimal mixed kernel that:

  1. Calls pl.system.set_ffts(workspace) on AIC and both AIV lanes.
  2. Stores an AIC matmul result to GM.
  3. Calls pl.system.sync_set(0, pipe=pl.PipeType.FIX, ffts_mode=2, core_type=pl.KernelType.AIC).
  4. Calls pl.system.sync_wait(0, pipe=pl.PipeType.MTE2, core_type=pl.KernelType.AIV) before each AIV lane reads its half.
  5. Compiles with platform="a2a3sim".

The failing downstream CI job is
https://github.com/hw-native-sys/pypto-lib/actions/runs/34444660251/job/102766782704.

Expected Behavior

The A2/A3 simulator should compile and execute explicit FFTS mode-2 events
with device-compatible synchronization semantics:

  • An AIC notification is visible to both AIV lanes.
  • An AIC wait completes only after both AIV lanes notify it.
  • Notifications are consuming credits, so queued and repeated events do not
    collapse into one boolean flag.
  • Event state is isolated by simulated device and cluster and reset between
    runs.

Actual Behavior

Generated simulator C++ fails to compile with errors including:

error: '__builtin_cce_wait_flag_dev' was not declared in this scope
error: 'FFTS_MODE_VAL' was not declared in this scope
error: 'getFFTSMsg' was not declared in this scope
error: '__builtin_cce_ffts_cross_core_sync' was not declared in this scope

CSA, HCA, and SWA all fail in this compilation phase. Native A2/A3 execution
passes, confirming that this is specific to the simulator compatibility path.

Git Commit ID

39ce891dbb3f665e72e99b4a1387b47012fb18bd

CANN Version

9.0.0

Driver Version

Not applicable to CPU simulation.

Host Platform

Linux (aarch64)

Additional Context

Proposed fix: #2186.

The fix supplies the emitted FFTS API on a2a3sim, uses the existing
runtime-owned per-device/per-cluster shared registry for cross-library event
credits, and forwards the existing simulator hooks into each independently
loaded kernel library. Unsupported modes/counts fail explicitly. Native and A5
compiler paths are unchanged.

Validation of the proposed fix includes the minimal two-AIV transfer golden,
cross-library event protocol tests, all five simulator scripts selected by the
downstream PR, and the runtime unit suite.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions